[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:12:00 dist-test-slave-9k0h krb5kdc[25449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:12:00 dist-test-slave-9k0h krb5kdc[25449](info): set up 2 sockets
Sep 29 02:12:00 dist-test-slave-9k0h krb5kdc[25449](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 02:12:01.822316 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:12:01 dist-test-slave-9k0h krb5kdc[25449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:12:01.992300 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:32881
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:32881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:02.326411 25462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:02.326689 25462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:02.326851 25462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:02.334323 25462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:02.334421 25462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:02.334507 25462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:02.334638 25462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:02.334719 25462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:12:02.344580 25462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:32881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:32881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:02.345868 25462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:02.347294 25462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:02.359679 25468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:02.359651 25467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:02.359704 25470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:02.360399 25462 server_base.cc:1034] running on GCE node
I20260929 02:12:02.361290 25462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:02.362742 25462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:02.364012 25462 hybrid_clock.cc:648] HybridClock initialized: now 1790647922363935 us; error 47 us; skew 500 ppm
I20260929 02:12:02.364391 25462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:02 dist-test-slave-9k0h krb5kdc[25449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647922, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:02.367600 25462 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:12:02.369175 25462 webserver.cc:460] Webserver started at http://127.24.216.190:38953/ using document root <none> and password file <none>
I20260929 02:12:02.369807 25462 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:02.369903 25462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:02.370194 25462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:02.373077 25462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "91f49012583f4c0192d1df6a103f4b10"
format_stamp: "Formatted at 2026-09-29 02:12:02 on dist-test-slave-9k0h"
server_key: "b965f917c586a43412d99a5b77c36023"
server_key_iv: "3018c467ec402ad15b64fb9ae2e98e2f"
server_key_version: "encryptionkey@0"
I20260929 02:12:02.373750 25462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "91f49012583f4c0192d1df6a103f4b10"
format_stamp: "Formatted at 2026-09-29 02:12:02 on dist-test-slave-9k0h"
server_key: "b965f917c586a43412d99a5b77c36023"
server_key_iv: "3018c467ec402ad15b64fb9ae2e98e2f"
server_key_version: "encryptionkey@0"
I20260929 02:12:02.379442 25462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.001s
I20260929 02:12:02.382635 25477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:02.383900 25462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:12:02.384027 25462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "91f49012583f4c0192d1df6a103f4b10"
format_stamp: "Formatted at 2026-09-29 02:12:02 on dist-test-slave-9k0h"
server_key: "b965f917c586a43412d99a5b77c36023"
server_key_iv: "3018c467ec402ad15b64fb9ae2e98e2f"
server_key_version: "encryptionkey@0"
I20260929 02:12:02.384184 25462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:02.403573 25462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:02.407383 25462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:02.407604 25462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:02.428131 25462 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:32881
I20260929 02:12:02.428131 25528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:32881 every 8 connection(s)
I20260929 02:12:02.429509 25462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:02.433351 25529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:02.438851 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 25462
I20260929 02:12:02.439188 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:02.441636 25442 external_mini_cluster.cc:1428] Setting key 934fd33defac8e1e38f3b0715de94a09
I20260929 02:12:02.446292 25529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10: Bootstrap starting.
I20260929 02:12:02.450376 25529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:02.451843 25529 log.cc:824] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:02.455446 25529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10: No bootstrap required, opened a new log
I20260929 02:12:02.467110 25529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f49012583f4c0192d1df6a103f4b10" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 32881 } }
I20260929 02:12:02.467528 25529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:02.467674 25529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f49012583f4c0192d1df6a103f4b10, State: Initialized, Role: FOLLOWER
I20260929 02:12:02.468295 25529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f49012583f4c0192d1df6a103f4b10" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 32881 } }
I20260929 02:12:02.468538 25529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:02.468685 25529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:02.468855 25529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:02.471150 25529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f49012583f4c0192d1df6a103f4b10" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 32881 } }
I20260929 02:12:02.471638 25529 leader_election.cc:302] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91f49012583f4c0192d1df6a103f4b10; no voters:
I20260929 02:12:02.472254 25529 leader_election.cc:288] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:02.472354 25534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:02.473676 25534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [term 1 LEADER]: Becoming Leader. State: Replica: 91f49012583f4c0192d1df6a103f4b10, State: Running, Role: LEADER
I20260929 02:12:02.474380 25534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f49012583f4c0192d1df6a103f4b10" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 32881 } }
I20260929 02:12:02.475112 25529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 02:12:02 dist-test-slave-9k0h krb5kdc[25449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:02.477839 25535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91f49012583f4c0192d1df6a103f4b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f49012583f4c0192d1df6a103f4b10" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 32881 } } }
I20260929 02:12:02.478312 25535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:02.482487 25536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91f49012583f4c0192d1df6a103f4b10. Latest consensus state: current_term: 1 leader_uuid: "91f49012583f4c0192d1df6a103f4b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f49012583f4c0192d1df6a103f4b10" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 32881 } } }
I20260929 02:12:02.482795 25536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:02.483870 25539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:02.489445 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:02.446143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.24.216.190:32881)
0929 02:12:02.447200 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:02.447218 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:02.448457 (+ 1239us) server_negotiation.cc:407] Connection header received
0929 02:12:02.461226 (+ 12769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:02.461248 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:02.461647 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:02.462032 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:02.466321 (+ 4289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:02.467730 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:02.469724 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:02.470514 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:02.470714 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:02.477454 (+ 6740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:02.477467 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:02.477489 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:02.477546 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:02.482432 (+ 4886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:02.483572 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:02.483583 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:02.483590 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:02.483772 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:02.484422 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:02.484430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:02.484434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:02.485160 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:02.485484 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:02.487024 (+ 1540us) server_negotiation.cc:299] Negotiation successful
0929 02:12:02.487620 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":465,"threads_started":1}
I20260929 02:12:02.490042 25539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:02.506227 25539 catalog_manager.cc:1347] Generated new cluster ID: a87cf3843040428dbcb41568cf06e37b
I20260929 02:12:02.506776 25539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:02.553398 25539 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:02.554894 25539 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:02.560110 25539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91f49012583f4c0192d1df6a103f4b10: Generated new TSK 0
I20260929 02:12:02.561105 25539 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:12:02.611541 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:32881
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:02.942755 25555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:02.943053 25555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:02.943215 25555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:02.950279 25555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:02.950423 25555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:02.950599 25555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:12:02.960834 25555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:32881
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:02.962221 25555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:02.963598 25555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:02.977222 25560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:02.977191 25561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:02.977120 25563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:02.978479 25555 server_base.cc:1034] running on GCE node
I20260929 02:12:02.979063 25555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:02.979967 25555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:02.981164 25555 hybrid_clock.cc:648] HybridClock initialized: now 1790647922981102 us; error 52 us; skew 500 ppm
I20260929 02:12:02.981491 25555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:02 dist-test-slave-9k0h krb5kdc[25449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647922, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:02.984651 25555 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:12:02.986250 25555 webserver.cc:460] Webserver started at http://127.24.216.129:33065/ using document root <none> and password file <none>
I20260929 02:12:02.986898 25555 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:02.987022 25555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:02.987332 25555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:02.990320 25555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "f47efb14ddc34477ad0769ec865c9400"
format_stamp: "Formatted at 2026-09-29 02:12:02 on dist-test-slave-9k0h"
server_key: "ee7f7b6829c8d2d9ecda7be4633c77c9"
server_key_iv: "0ef5781dde1baca4f1c70f4cec908e80"
server_key_version: "encryptionkey@0"
I20260929 02:12:02.991035 25555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "f47efb14ddc34477ad0769ec865c9400"
format_stamp: "Formatted at 2026-09-29 02:12:02 on dist-test-slave-9k0h"
server_key: "ee7f7b6829c8d2d9ecda7be4633c77c9"
server_key_iv: "0ef5781dde1baca4f1c70f4cec908e80"
server_key_version: "encryptionkey@0"
I20260929 02:12:02.996052 25555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:12:02.999915 25570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:03.001231 25555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 02:12:03.001416 25555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "f47efb14ddc34477ad0769ec865c9400"
format_stamp: "Formatted at 2026-09-29 02:12:02 on dist-test-slave-9k0h"
server_key: "ee7f7b6829c8d2d9ecda7be4633c77c9"
server_key_iv: "0ef5781dde1baca4f1c70f4cec908e80"
server_key_version: "encryptionkey@0"
I20260929 02:12:03.001637 25555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:03.021448 25555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:03.025240 25555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:03.025633 25555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:03.026621 25555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:03.027858 25555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:03.027959 25555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:03.028106 25555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:03.028179 25555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:03.064877 25555 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:36643
I20260929 02:12:03.064925 25682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:36643 every 8 connection(s)
I20260929 02:12:03.066234 25555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:12:03.069013 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 25555
I20260929 02:12:03.069298 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:12:03.070039 25442 external_mini_cluster.cc:1428] Setting key c455514203e2f8f3c6f051ce49165de3
Sep 29 02:12:03 dist-test-slave-9k0h krb5kdc[25449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647922, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
I20260929 02:12:03.089126 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:03.068861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:37689 (local address 127.24.216.190:32881)
0929 02:12:03.069810 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:03.069817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:03.070850 (+ 1033us) server_negotiation.cc:407] Connection header received
0929 02:12:03.072711 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:03.072719 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:03.072811 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:03.072950 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:03.074426 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:03.075222 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:03.077271 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:03.077972 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:03.078128 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:03.081673 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:03.081682 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:03.081689 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:03.081732 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:03.085427 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:03.086226 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:03.086232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:03.086236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:03.086380 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:03.086788 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:03.086796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:03.086801 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:03.087114 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:03.087276 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:03.088039 (+ 763us) server_negotiation.cc:299] Negotiation successful
0929 02:12:03.088194 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":702,"threads_started":1}
I20260929 02:12:03.089553 25686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:03.069859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:32881 (local address 127.24.216.129:37689)
0929 02:12:03.070648 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:03.070700 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:03.072442 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:03.073145 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:03.073153 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:03.073649 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:03.074268 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:03.074286 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:03.075401 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:03.075409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:03.077133 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:03.077144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:03.078252 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:03.078257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:03.078447 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:03.079117 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:03.079142 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:03.081451 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:03.085556 (+ 4105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:03.085563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:03.085575 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:03.086075 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:03.086490 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:03.086493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:03.086495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:03.086659 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:03.087233 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:03.087239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:03.087642 (+ 403us) client_negotiation.cc:769] Sending connection context
0929 02:12:03.087943 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 02:12:03.088369 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":339,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
I20260929 02:12:03.091176 25683 heartbeater.cc:347] Connected to a master server at 127.24.216.190:32881
I20260929 02:12:03.091460 25683 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:03.092144 25683 heartbeater.cc:511] Master 127.24.216.190:32881 requested a full tablet report, sending...
I20260929 02:12:03.094120 25494 ts_manager.cc:194] Registered new tserver with Master: f47efb14ddc34477ad0769ec865c9400 (127.24.216.129:36643)
I20260929 02:12:03.095574 25494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:37689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:12:03.112008 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:32881
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:03.446364 25689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:03.446681 25689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:03.446841 25689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:03.453933 25689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:03.454070 25689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:03.454269 25689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:12:03.463573 25689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:32881
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:03.464816 25689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:03.466259 25689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:03.479635 25694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:03.479696 25695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:03.480336 25697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:03.480595 25689 server_base.cc:1034] running on GCE node
I20260929 02:12:03.481163 25689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:03.482043 25689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:03.483276 25689 hybrid_clock.cc:648] HybridClock initialized: now 1790647923483248 us; error 62 us; skew 500 ppm
I20260929 02:12:03.483570 25689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:03 dist-test-slave-9k0h krb5kdc[25449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:03.487210 25689 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:12:03.488799 25689 webserver.cc:460] Webserver started at http://127.24.216.130:41605/ using document root <none> and password file <none>
I20260929 02:12:03.489472 25689 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:03.489619 25689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:03.489926 25689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:03.492908 25689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "efa8a6cd9cfd455a816c7eb56d4edec7"
format_stamp: "Formatted at 2026-09-29 02:12:03 on dist-test-slave-9k0h"
server_key: "48d9e2d27a6fad4b552f8b27be68d4ca"
server_key_iv: "bf55ddc2161d52cd23460a62a2f2ff02"
server_key_version: "encryptionkey@0"
I20260929 02:12:03.493688 25689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "efa8a6cd9cfd455a816c7eb56d4edec7"
format_stamp: "Formatted at 2026-09-29 02:12:03 on dist-test-slave-9k0h"
server_key: "48d9e2d27a6fad4b552f8b27be68d4ca"
server_key_iv: "bf55ddc2161d52cd23460a62a2f2ff02"
server_key_version: "encryptionkey@0"
I20260929 02:12:03.498865 25689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 02:12:03.502054 25704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:03.503364 25689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:12:03.503537 25689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "efa8a6cd9cfd455a816c7eb56d4edec7"
format_stamp: "Formatted at 2026-09-29 02:12:03 on dist-test-slave-9k0h"
server_key: "48d9e2d27a6fad4b552f8b27be68d4ca"
server_key_iv: "bf55ddc2161d52cd23460a62a2f2ff02"
server_key_version: "encryptionkey@0"
I20260929 02:12:03.503751 25689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:03.521189 25689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:03.525532 25689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:03.525825 25689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:03.526772 25689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:03.528074 25689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:03.528174 25689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:03.528357 25689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:03.528426 25689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:03.564217 25689 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:38177
I20260929 02:12:03.564304 25816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:38177 every 8 connection(s)
I20260929 02:12:03.565531 25689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:12:03.572181 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 25689
I20260929 02:12:03.572382 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:12:03.572768 25442 external_mini_cluster.cc:1428] Setting key 62f3c8f8504587617f05a10d9442fee0
Sep 29 02:12:03 dist-test-slave-9k0h krb5kdc[25449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:03.586043 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:03.568113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:43567 (local address 127.24.216.190:32881)
0929 02:12:03.568279 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:03.568283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:03.569904 (+ 1621us) server_negotiation.cc:407] Connection header received
0929 02:12:03.571855 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:03.571860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:03.571944 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:03.572056 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:03.573575 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:03.574517 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:03.576169 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:03.576691 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:03.576855 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:03.580360 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:03.580372 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:03.580375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:03.580406 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:03.582952 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:03.583644 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:03.583651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:03.583654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:03.583782 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:03.584205 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:03.584211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:03.584215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:03.584527 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:03.584690 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:03.585597 (+ 907us) server_negotiation.cc:299] Negotiation successful
0929 02:12:03.585761 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:12:03.587764 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:03.568742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:32881 (local address 127.24.216.130:43567)
0929 02:12:03.569741 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:03.569783 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:03.571581 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:03.572290 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:03.572298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:03.572873 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:03.573415 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:03.573431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:03.574671 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:03.574675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:03.576029 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:03.576040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:03.576941 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:03.576948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:03.577163 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:03.577853 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:03.577877 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:03.580143 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:03.583089 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:03.583097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:03.583113 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:03.583525 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:03.583891 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:03.583897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:03.583900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:03.584113 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:03.584687 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:03.584699 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:03.585170 (+ 471us) client_negotiation.cc:769] Sending connection context
0929 02:12:03.585519 (+ 349us) client_negotiation.cc:240] Negotiation successful
0929 02:12:03.586203 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":611,"threads_started":1}
I20260929 02:12:03.589363 25817 heartbeater.cc:347] Connected to a master server at 127.24.216.190:32881
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
I20260929 02:12:03.589684 25817 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:03.590541 25817 heartbeater.cc:511] Master 127.24.216.190:32881 requested a full tablet report, sending...
I20260929 02:12:03.591964 25494 ts_manager.cc:194] Registered new tserver with Master: efa8a6cd9cfd455a816c7eb56d4edec7 (127.24.216.130:38177)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
I20260929 02:12:03.593137 25494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:43567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:12:03.611723 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:32881
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:03.932561 25822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:03.932847 25822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:03.932940 25822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:03.939952 25822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:03.940045 25822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:03.940162 25822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:12:03.949523 25822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:32881
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:03.950659 25822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:03.951829 25822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:03.964200 25827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:03.964233 25828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:03.964282 25830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:03.965431 25822 server_base.cc:1034] running on GCE node
I20260929 02:12:03.965953 25822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:03.966894 25822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:03.968075 25822 hybrid_clock.cc:648] HybridClock initialized: now 1790647923968020 us; error 56 us; skew 500 ppm
I20260929 02:12:03.968366 25822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:03 dist-test-slave-9k0h krb5kdc[25449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:03.971377 25822 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:12:03.972750 25822 webserver.cc:460] Webserver started at http://127.24.216.131:33419/ using document root <none> and password file <none>
I20260929 02:12:03.973381 25822 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:03.973505 25822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:03.973793 25822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:03.976691 25822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "6dcab21ee035461fb69a643aabdf1eb7"
format_stamp: "Formatted at 2026-09-29 02:12:03 on dist-test-slave-9k0h"
server_key: "ace80f9fa80298a2dea17521acbab53c"
server_key_iv: "51120ec6f68b99a8f3969a64664f0411"
server_key_version: "encryptionkey@0"
I20260929 02:12:03.977391 25822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "6dcab21ee035461fb69a643aabdf1eb7"
format_stamp: "Formatted at 2026-09-29 02:12:03 on dist-test-slave-9k0h"
server_key: "ace80f9fa80298a2dea17521acbab53c"
server_key_iv: "51120ec6f68b99a8f3969a64664f0411"
server_key_version: "encryptionkey@0"
I20260929 02:12:03.982611 25822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 02:12:03.985927 25837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:03.987206 25822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:03.987385 25822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "6dcab21ee035461fb69a643aabdf1eb7"
format_stamp: "Formatted at 2026-09-29 02:12:03 on dist-test-slave-9k0h"
server_key: "ace80f9fa80298a2dea17521acbab53c"
server_key_iv: "51120ec6f68b99a8f3969a64664f0411"
server_key_version: "encryptionkey@0"
I20260929 02:12:03.987571 25822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:04.008762 25822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:04.012681 25822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:04.012996 25822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:04.013922 25822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:04.015264 25822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:04.015383 25822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:04.015549 25822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:04.015621 25822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:04.053524 25822 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:40421
I20260929 02:12:04.053567 25949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:40421 every 8 connection(s)
I20260929 02:12:04.054865 25822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:12:04.057328 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 25822
I20260929 02:12:04.057519 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:12:04.058028 25442 external_mini_cluster.cc:1428] Setting key 86c225b58228b288f48b5f0b86909f16
Sep 29 02:12:04 dist-test-slave-9k0h krb5kdc[25449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647923, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:04.077047 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:04.057623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:33099 (local address 127.24.216.190:32881)
0929 02:12:04.057783 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:04.057787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:04.059272 (+ 1485us) server_negotiation.cc:407] Connection header received
0929 02:12:04.061637 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:04.061641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:04.061698 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:04.061813 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:04.063365 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:04.064177 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:04.066394 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:04.066804 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:04.066899 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:04.071229 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:04.071234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:04.071237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:04.071266 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:04.074014 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:04.074805 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:04.074809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:04.074811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:04.074923 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:04.075323 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:04.075331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:04.075334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:04.075630 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:04.075793 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:04.076497 (+ 704us) server_negotiation.cc:299] Negotiation successful
0929 02:12:04.076712 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 02:12:04.078011 25952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:04.058278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:32881 (local address 127.24.216.131:33099)
0929 02:12:04.059083 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:04.059138 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:04.061303 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:04.062021 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:04.062028 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:04.062669 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:04.063193 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:04.063210 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:04.064321 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:04.064327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:04.066257 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:04.066279 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:04.067010 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:04.067017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:04.067284 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:04.067938 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:04.067964 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:04.071029 (+ 3065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:04.074182 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:04.074193 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:04.074207 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:04.074678 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:04.075014 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:04.075017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:04.075019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:04.075184 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:04.075764 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:04.075772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:04.076150 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 02:12:04.076396 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 02:12:04.076904 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":299,"threads_started":1}
I20260929 02:12:04.079288 25950 heartbeater.cc:347] Connected to a master server at 127.24.216.190:32881
I20260929 02:12:04.079553 25950 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:04.080237 25950 heartbeater.cc:511] Master 127.24.216.190:32881 requested a full tablet report, sending...
I20260929 02:12:04.081452 25494 ts_manager.cc:194] Registered new tserver with Master: 6dcab21ee035461fb69a643aabdf1eb7 (127.24.216.131:40421)
I20260929 02:12:04.082274 25494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:33099
I20260929 02:12:04.086462 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:12:04.098842 25683 heartbeater.cc:503] Master 127.24.216.190:32881 was elected leader, sending a full tablet report...
Sep 29 02:12:04 dist-test-slave-9k0h krb5kdc[25449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:12:04 dist-test-slave-9k0h krb5kdc[25449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:12:04.132499 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:04.116514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51340 (local address 127.24.216.129:36643)
0929 02:12:04.117264 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:04.117275 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:04.117383 (+ 108us) server_negotiation.cc:407] Connection header received
0929 02:12:04.117621 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:04.117638 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:04.117869 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:04.118056 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:04.120196 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:04.121246 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:04.122686 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:04.123283 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:04.123443 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:04.127652 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:04.127663 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:04.127678 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:04.127707 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:04.130340 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:04.130884 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:04.130891 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:04.130893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:04.130999 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:04.131297 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:04.131300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:04.131302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:04.131603 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:04.131814 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:04.132041 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 02:12:04.132231 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":336,"threads_started":1}
W20260929 02:12:04.133701 25598 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51340
Sep 29 02:12:04 dist-test-slave-9k0h krb5kdc[25449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:12:04 dist-test-slave-9k0h krb5kdc[25449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:12:04.178007 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:04.163723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.24.216.129:36643)
0929 02:12:04.163973 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:04.163980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:04.164281 (+ 301us) server_negotiation.cc:407] Connection header received
0929 02:12:04.164515 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:04.164528 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:04.164634 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:04.164755 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:04.166210 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:04.166951 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:04.168241 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:04.168691 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:04.168839 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:04.172930 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:04.172935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:04.172938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:04.172969 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:04.175688 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:04.176245 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:04.176249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:04.176251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:04.176363 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:04.176710 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:04.176716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:04.176720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:04.176961 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:04.177210 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:04.177487 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 02:12:04.177693 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 02:12:04.179648 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 25555
I20260929 02:12:04.192410 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:04.180745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.24.216.129:36643)
0929 02:12:04.180938 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:04.180945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:04.181065 (+ 120us) server_negotiation.cc:407] Connection header received
0929 02:12:04.181182 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:04.181186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:04.181240 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:04.181338 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:04.182600 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:04.183388 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:04.184626 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:04.185119 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:04.185224 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:04.187512 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:04.187519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:04.187523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:04.187560 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:04.190114 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:04.190721 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:04.190725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:04.190727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:04.190854 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:04.191168 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:04.191172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:04.191175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:04.191426 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:04.191597 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:04.191899 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 02:12:04.192041 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:12:04.193141 25678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:04.275802 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 25555
I20260929 02:12:04.293435 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 25689
Sep 29 02:12:04 dist-test-slave-9k0h krb5kdc[25449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:12:04.308758 25964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:04.294333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.24.216.130:38177)
0929 02:12:04.294859 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:04.294866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:04.294890 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:12:04.294953 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:04.294959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:04.295168 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:04.295344 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:04.296506 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:04.297454 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:04.298827 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:04.299382 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:04.299525 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:04.303435 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:04.303447 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:04.303460 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:04.303489 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:04.305957 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:04.306684 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:04.306691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:04.306693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:04.306820 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:04.307229 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:04.307233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:04.307235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:04.307541 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:04.307849 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:04.308167 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 02:12:04.308421 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":255,"threads_started":1}
I20260929 02:12:04.309996 25812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:04.390367 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 25689
I20260929 02:12:04.412804 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 25822
Sep 29 02:12:04 dist-test-slave-9k0h krb5kdc[25449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:12:04.428231 25966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:04.413767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.24.216.131:40421)
0929 02:12:04.414410 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:04.414417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:04.414445 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:12:04.414501 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:04.414508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:04.414734 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:04.414855 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:04.415980 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:04.416987 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:04.418211 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:04.418791 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:04.418884 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:04.422878 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:04.422888 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:04.422901 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:04.422933 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:04.425689 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:04.426366 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:04.426377 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:04.426381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:04.426505 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:04.426817 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:04.426820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:04.426822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:04.427116 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:04.427338 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:04.427649 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 02:12:04.427909 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":304,"threads_started":1}
I20260929 02:12:04.429481 25945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:04.511574 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 25822
I20260929 02:12:04.527983 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 25462
I20260929 02:12:04.529102 25524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:04.630127 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 25462
2026-09-29T02:12:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3913 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:12:04 dist-test-slave-9k0h krb5kdc[25971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:12:04 dist-test-slave-9k0h krb5kdc[25971](info): set up 2 sockets
Sep 29 02:12:04 dist-test-slave-9k0h krb5kdc[25971](info): commencing operation
krb5kdc: starting...
W20260929 02:12:05.698223 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:12:05 dist-test-slave-9k0h krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:12:05.867671 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:37005
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:37005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:06.171756 25984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:06.172087 25984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:06.172266 25984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:06.179256 25984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:06.179405 25984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:06.179510 25984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:06.179610 25984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:06.179695 25984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:12:06.188980 25984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:37005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:37005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:06.190284 25984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:06.191587 25984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:06.203521 25989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:06.203569 25990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:06.203846 25984 server_base.cc:1034] running on GCE node
W20260929 02:12:06.203588 25992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:06.204769 25984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:06.206202 25984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:06.207454 25984 hybrid_clock.cc:648] HybridClock initialized: now 1790647926207409 us; error 30 us; skew 500 ppm
I20260929 02:12:06.207778 25984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:06 dist-test-slave-9k0h krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647926, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:06.220449 25984 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:12:06.221889 25984 webserver.cc:460] Webserver started at http://127.24.216.190:44995/ using document root <none> and password file <none>
I20260929 02:12:06.222625 25984 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:06.222777 25984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:06.223088 25984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:06.225898 25984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "e017f8c3d7f14102a314099a8be767e2"
format_stamp: "Formatted at 2026-09-29 02:12:06 on dist-test-slave-9k0h"
server_key: "732121ce6561e63b905bed5e6445729b"
server_key_iv: "90d9eaef9e06a244fb1f828b4b7b30e4"
server_key_version: "encryptionkey@0"
I20260929 02:12:06.226675 25984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "e017f8c3d7f14102a314099a8be767e2"
format_stamp: "Formatted at 2026-09-29 02:12:06 on dist-test-slave-9k0h"
server_key: "732121ce6561e63b905bed5e6445729b"
server_key_iv: "90d9eaef9e06a244fb1f828b4b7b30e4"
server_key_version: "encryptionkey@0"
I20260929 02:12:06.231613 25984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:12:06.234805 25999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:06.235970 25984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 02:12:06.236145 25984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "e017f8c3d7f14102a314099a8be767e2"
format_stamp: "Formatted at 2026-09-29 02:12:06 on dist-test-slave-9k0h"
server_key: "732121ce6561e63b905bed5e6445729b"
server_key_iv: "90d9eaef9e06a244fb1f828b4b7b30e4"
server_key_version: "encryptionkey@0"
I20260929 02:12:06.236330 25984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:06.248229 25984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:06.251863 25984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:06.252130 25984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:06.269609 25984 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:37005
I20260929 02:12:06.269616 26050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:37005 every 8 connection(s)
I20260929 02:12:06.270983 25984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:06.272542 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 25984
I20260929 02:12:06.272778 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:06.273419 25442 external_mini_cluster.cc:1428] Setting key 590b0be44f4bcc11ba71c7744e6f58b1
I20260929 02:12:06.274996 26051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:12:06 dist-test-slave-9k0h krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:06.289433 26051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2: Bootstrap starting.
I20260929 02:12:06.293699 26051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:06.295006 26051 log.cc:824] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:06.298255 26051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2: No bootstrap required, opened a new log
I20260929 02:12:06.298415 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:06.275423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.24.216.190:37005)
0929 02:12:06.276157 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:06.276172 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:06.276221 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:12:06.278215 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:06.278242 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:06.278759 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:06.279194 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:06.280450 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:06.282528 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:06.283711 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:06.284702 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:06.284926 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:06.289688 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:06.289698 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:06.289721 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:06.289780 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:06.293760 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:06.294624 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:06.294631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:06.294636 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:06.294799 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:06.295234 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:06.295239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:06.295242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:06.295908 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:06.296128 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:06.296747 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 02:12:06.297165 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":309,"threads_started":1}
I20260929 02:12:06.306018 26051 raft_consensus.cc:348] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e017f8c3d7f14102a314099a8be767e2" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37005 } }
I20260929 02:12:06.306411 26051 raft_consensus.cc:374] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:06.306486 26051 raft_consensus.cc:729] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e017f8c3d7f14102a314099a8be767e2, State: Initialized, Role: FOLLOWER
I20260929 02:12:06.306931 26051 consensus_queue.cc:260] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e017f8c3d7f14102a314099a8be767e2" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37005 } }
I20260929 02:12:06.307113 26051 raft_consensus.cc:388] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:06.307199 26051 raft_consensus.cc:482] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:06.307328 26051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:06.309602 26051 raft_consensus.cc:504] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e017f8c3d7f14102a314099a8be767e2" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37005 } }
I20260929 02:12:06.310005 26051 leader_election.cc:302] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e017f8c3d7f14102a314099a8be767e2; no voters:
I20260929 02:12:06.310583 26051 leader_election.cc:288] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:06.310716 26056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:06.311501 26056 raft_consensus.cc:686] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [term 1 LEADER]: Becoming Leader. State: Replica: e017f8c3d7f14102a314099a8be767e2, State: Running, Role: LEADER
I20260929 02:12:06.312029 26056 consensus_queue.cc:237] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e017f8c3d7f14102a314099a8be767e2" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37005 } }
I20260929 02:12:06.312638 26051 sys_catalog.cc:564] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:06.314881 26057 sys_catalog.cc:455] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e017f8c3d7f14102a314099a8be767e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e017f8c3d7f14102a314099a8be767e2" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37005 } } }
I20260929 02:12:06.314890 26058 sys_catalog.cc:455] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e017f8c3d7f14102a314099a8be767e2. Latest consensus state: current_term: 1 leader_uuid: "e017f8c3d7f14102a314099a8be767e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e017f8c3d7f14102a314099a8be767e2" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37005 } } }
I20260929 02:12:06.315256 26057 sys_catalog.cc:458] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:06.315326 26058 sys_catalog.cc:458] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:06.317317 26062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:06.322036 26062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:06.329834 26062 catalog_manager.cc:1347] Generated new cluster ID: cc3f28755a5b4d1f8433ba8da719c25d
I20260929 02:12:06.329948 26062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:06.348349 26062 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:06.350188 26062 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:06.356726 26062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e017f8c3d7f14102a314099a8be767e2: Generated new TSK 0
I20260929 02:12:06.357708 26062 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:12:06.398872 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37005
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:06.715704 26077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:06.715981 26077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:06.716075 26077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:06.723305 26077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:06.723403 26077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:06.723534 26077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:12:06.733023 26077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37005
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:06.734318 26077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:06.735556 26077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:06.748504 26082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:06.748561 26083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:06.748915 26085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:06.749248 26077 server_base.cc:1034] running on GCE node
I20260929 02:12:06.749830 26077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:06.750795 26077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:06.752007 26077 hybrid_clock.cc:648] HybridClock initialized: now 1790647926751967 us; error 52 us; skew 500 ppm
I20260929 02:12:06.752327 26077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:06 dist-test-slave-9k0h krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647926, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:06.755545 26077 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:12:06.756897 26077 webserver.cc:460] Webserver started at http://127.24.216.129:46237/ using document root <none> and password file <none>
I20260929 02:12:06.757565 26077 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:06.757694 26077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:06.758009 26077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:06.760870 26077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "0f1a517465344dacaafa33439fcf01a4"
format_stamp: "Formatted at 2026-09-29 02:12:06 on dist-test-slave-9k0h"
server_key: "8b0ffa8ba4f8e4c5a1d3b45b240e29f1"
server_key_iv: "f73214170eb74b7eb060859d246c3064"
server_key_version: "encryptionkey@0"
I20260929 02:12:06.761643 26077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "0f1a517465344dacaafa33439fcf01a4"
format_stamp: "Formatted at 2026-09-29 02:12:06 on dist-test-slave-9k0h"
server_key: "8b0ffa8ba4f8e4c5a1d3b45b240e29f1"
server_key_iv: "f73214170eb74b7eb060859d246c3064"
server_key_version: "encryptionkey@0"
I20260929 02:12:06.766563 26077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:12:06.769788 26092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:06.771101 26077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:06.771288 26077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "0f1a517465344dacaafa33439fcf01a4"
format_stamp: "Formatted at 2026-09-29 02:12:06 on dist-test-slave-9k0h"
server_key: "8b0ffa8ba4f8e4c5a1d3b45b240e29f1"
server_key_iv: "f73214170eb74b7eb060859d246c3064"
server_key_version: "encryptionkey@0"
I20260929 02:12:06.771497 26077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:06.806572 26077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:06.810407 26077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:06.810685 26077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:06.811597 26077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:06.812907 26077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:06.813005 26077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:06.813150 26077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:06.813220 26077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:06.845911 26077 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:42009
I20260929 02:12:06.845945 26204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:42009 every 8 connection(s)
I20260929 02:12:06.847215 26077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:12:06.855197 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 26077
I20260929 02:12:06.855413 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:12:06.856110 25442 external_mini_cluster.cc:1428] Setting key a125d0a18ed2ceef8bf99e710e2403db
Sep 29 02:12:06 dist-test-slave-9k0h krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647926, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:06.866688 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:06.849636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:49913 (local address 127.24.216.190:37005)
0929 02:12:06.850068 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:06.850075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:06.851048 (+ 973us) server_negotiation.cc:407] Connection header received
0929 02:12:06.852695 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:06.852699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:06.852761 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:06.852860 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:06.854238 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:06.854959 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:06.856767 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:06.857324 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:06.857473 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:06.861247 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:06.861252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:06.861255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:06.861291 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:06.863778 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:06.864461 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:06.864466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:06.864470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:06.864585 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:06.864994 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:06.864999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:06.865002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:06.865283 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:06.865428 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:06.866085 (+ 657us) server_negotiation.cc:299] Negotiation successful
0929 02:12:06.866345 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":207,"threads_started":1}
I20260929 02:12:06.867605 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:06.850267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:37005 (local address 127.24.216.129:49913)
0929 02:12:06.850901 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:06.850938 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:06.852461 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:06.853061 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:06.853069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:06.853549 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:06.854061 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:06.854076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:06.855114 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:06.855124 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:06.856610 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:06.856621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:06.857552 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:06.857559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:06.857848 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:06.858575 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:06.858599 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:06.861039 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:06.863917 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:06.863926 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:06.863942 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:06.864336 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:06.864691 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:06.864698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:06.864701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:06.864879 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:06.865418 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:06.865425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:06.865738 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 02:12:06.866000 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 02:12:06.866540 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":245,"threads_started":1}
I20260929 02:12:06.869093 26205 heartbeater.cc:347] Connected to a master server at 127.24.216.190:37005
I20260929 02:12:06.869391 26205 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:06.870103 26205 heartbeater.cc:511] Master 127.24.216.190:37005 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
I20260929 02:12:06.872483 26016 ts_manager.cc:194] Registered new tserver with Master: 0f1a517465344dacaafa33439fcf01a4 (127.24.216.129:42009)
I20260929 02:12:06.873943 26016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:49913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:12:06.895272 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37005
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:07.204906 26211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:07.205152 26211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:07.205353 26211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:07.212503 26211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:07.212668 26211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:07.212857 26211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:12:07.224694 26211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37005
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:07.226042 26211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:07.227375 26211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:07.240235 26216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:07.240311 26219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:07.240345 26217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:07.241380 26211 server_base.cc:1034] running on GCE node
I20260929 02:12:07.241940 26211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:07.242866 26211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:07.244093 26211 hybrid_clock.cc:648] HybridClock initialized: now 1790647927244055 us; error 48 us; skew 500 ppm
I20260929 02:12:07.244391 26211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:07 dist-test-slave-9k0h krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647927, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:07.247574 26211 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:12:07.248854 26211 webserver.cc:460] Webserver started at http://127.24.216.130:38717/ using document root <none> and password file <none>
I20260929 02:12:07.249490 26211 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:07.249629 26211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:07.249926 26211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:07.252651 26211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "6c992f5d7369487f9747e7b4ba0bfb2b"
format_stamp: "Formatted at 2026-09-29 02:12:07 on dist-test-slave-9k0h"
server_key: "3a817803107b67708d76e9d21968965e"
server_key_iv: "19c5a80f4a373af966e5e1dbb5f8b293"
server_key_version: "encryptionkey@0"
I20260929 02:12:07.253389 26211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "6c992f5d7369487f9747e7b4ba0bfb2b"
format_stamp: "Formatted at 2026-09-29 02:12:07 on dist-test-slave-9k0h"
server_key: "3a817803107b67708d76e9d21968965e"
server_key_iv: "19c5a80f4a373af966e5e1dbb5f8b293"
server_key_version: "encryptionkey@0"
I20260929 02:12:07.258109 26211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 02:12:07.261379 26226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:07.262746 26211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:07.262919 26211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "6c992f5d7369487f9747e7b4ba0bfb2b"
format_stamp: "Formatted at 2026-09-29 02:12:07 on dist-test-slave-9k0h"
server_key: "3a817803107b67708d76e9d21968965e"
server_key_iv: "19c5a80f4a373af966e5e1dbb5f8b293"
server_key_version: "encryptionkey@0"
I20260929 02:12:07.263109 26211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:07.299149 26211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:07.303411 26211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:07.303686 26211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:07.304620 26211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:07.305888 26211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:07.305991 26211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:07.306139 26211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:07.306262 26211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 02:12:07.337963 26211 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:45263
I20260929 02:12:07.338016 26338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:45263 every 8 connection(s)
I20260929 02:12:07.339314 26211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:12:07.340929 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 26211
I20260929 02:12:07.341167 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:12:07.341703 25442 external_mini_cluster.cc:1428] Setting key 10ab52293a514d5aa75cc3f83342bc74
Sep 29 02:12:07 dist-test-slave-9k0h krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647927, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
I20260929 02:12:07.360033 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:07.341879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:38301 (local address 127.24.216.190:37005)
0929 02:12:07.342090 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:07.342096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:07.343448 (+ 1352us) server_negotiation.cc:407] Connection header received
0929 02:12:07.345264 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:07.345269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:07.345359 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:07.345476 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:07.347111 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.347968 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.349549 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.350391 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.350526 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:07.353587 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:07.353596 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:07.353603 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:07.353646 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:07.357200 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:07.357881 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:07.357889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:07.357895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:07.358049 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:07.358543 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:07.358546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:07.358548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:07.358803 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:07.358957 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:07.359631 (+ 674us) server_negotiation.cc:299] Negotiation successful
0929 02:12:07.359794 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
I20260929 02:12:07.361071 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:07.342444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:37005 (local address 127.24.216.130:38301)
0929 02:12:07.343293 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:07.343352 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:07.345004 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:07.345711 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:07.345724 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:07.346406 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:07.346954 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:07.346969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.348122 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.348126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:07.349407 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:07.349418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.350573 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.350578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:07.350755 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:07.351405 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:07.351427 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:07.353377 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:07.357350 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:07.357356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:07.357368 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:07.357767 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:07.358195 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:07.358203 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:07.358206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:07.358391 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:07.358928 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:07.358936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:07.359243 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 02:12:07.359542 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 02:12:07.359904 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":333,"threads_started":1}
I20260929 02:12:07.362563 26339 heartbeater.cc:347] Connected to a master server at 127.24.216.190:37005
I20260929 02:12:07.362872 26339 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:07.363575 26339 heartbeater.cc:511] Master 127.24.216.190:37005 requested a full tablet report, sending...
I20260929 02:12:07.365099 26016 ts_manager.cc:194] Registered new tserver with Master: 6c992f5d7369487f9747e7b4ba0bfb2b (127.24.216.130:45263)
I20260929 02:12:07.366350 26016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:38301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:12:07.383526 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37005
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:07.691944 26344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:07.692214 26344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:07.692327 26344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:07.699448 26344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:07.699555 26344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:07.699669 26344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:12:07.709168 26344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37005
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:07.710433 26344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:07.711706 26344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:07.724833 26349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:07.724781 26352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:07.724814 26350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:07.725639 26344 server_base.cc:1034] running on GCE node
I20260929 02:12:07.726336 26344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:07.727293 26344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:07.728529 26344 hybrid_clock.cc:648] HybridClock initialized: now 1790647927728459 us; error 34 us; skew 500 ppm
I20260929 02:12:07.728880 26344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:07 dist-test-slave-9k0h krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647927, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:07.731981 26344 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:12:07.733460 26344 webserver.cc:460] Webserver started at http://127.24.216.131:35571/ using document root <none> and password file <none>
I20260929 02:12:07.734138 26344 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:07.734334 26344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:07.734653 26344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:07.737591 26344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "0ee38cb6bc634a5ca42306864554f497"
format_stamp: "Formatted at 2026-09-29 02:12:07 on dist-test-slave-9k0h"
server_key: "74725c01836ff06e3602fb07e26f4390"
server_key_iv: "869bd8af9438ee0b5e3dddf2fb4c62b0"
server_key_version: "encryptionkey@0"
I20260929 02:12:07.738420 26344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "0ee38cb6bc634a5ca42306864554f497"
format_stamp: "Formatted at 2026-09-29 02:12:07 on dist-test-slave-9k0h"
server_key: "74725c01836ff06e3602fb07e26f4390"
server_key_iv: "869bd8af9438ee0b5e3dddf2fb4c62b0"
server_key_version: "encryptionkey@0"
I20260929 02:12:07.743716 26344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:12:07.747133 26359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:07.748466 26344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:12:07.748642 26344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "0ee38cb6bc634a5ca42306864554f497"
format_stamp: "Formatted at 2026-09-29 02:12:07 on dist-test-slave-9k0h"
server_key: "74725c01836ff06e3602fb07e26f4390"
server_key_iv: "869bd8af9438ee0b5e3dddf2fb4c62b0"
server_key_version: "encryptionkey@0"
I20260929 02:12:07.748842 26344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:07.768533 26344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:07.772552 26344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:07.772850 26344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:07.773790 26344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:07.775193 26344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:07.775264 26344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:07.775436 26344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:07.775509 26344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:07.808288 26344 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:35675
I20260929 02:12:07.808406 26471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:35675 every 8 connection(s)
I20260929 02:12:07.809625 26344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:12:07.818416 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 26344
I20260929 02:12:07.818621 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:12:07.819164 25442 external_mini_cluster.cc:1428] Setting key 5e58762ba945da441c28d12dc84569ba
Sep 29 02:12:07 dist-test-slave-9k0h krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647927, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:07.830363 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:07.812279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:35553 (local address 127.24.216.190:37005)
0929 02:12:07.812476 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:07.812480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:07.814385 (+ 1905us) server_negotiation.cc:407] Connection header received
0929 02:12:07.816225 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:07.816231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:07.816407 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:07.816532 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:07.818087 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.819063 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.821190 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.821804 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.821936 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:07.825245 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:07.825250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:07.825253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:07.825284 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:07.827776 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:07.828449 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:07.828452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:07.828454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:07.828560 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:07.828911 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:07.828914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:07.828916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:07.829187 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:07.829306 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:07.829887 (+ 581us) server_negotiation.cc:299] Negotiation successful
0929 02:12:07.830031 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:12:07.831348 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:07.812863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:37005 (local address 127.24.216.131:35553)
0929 02:12:07.814229 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:07.814268 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:07.815945 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:07.816788 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:07.816797 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:07.817332 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:07.817892 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:07.817909 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.819234 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.819241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:07.821029 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:07.821046 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.822034 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.822043 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:07.822358 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:07.822950 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:07.822976 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:07.825041 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:07.827902 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:07.827909 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:07.827921 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:07.828353 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:07.828649 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:07.828652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:07.828654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:07.828817 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:07.829296 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:07.829301 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:07.829592 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 02:12:07.829816 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 02:12:07.830268 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"thread_start_us":796,"threads_started":1}
I20260929 02:12:07.832659 26472 heartbeater.cc:347] Connected to a master server at 127.24.216.190:37005
I20260929 02:12:07.833115 26472 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:07.834022 26472 heartbeater.cc:511] Master 127.24.216.190:37005 requested a full tablet report, sending...
I20260929 02:12:07.835649 26016 ts_manager.cc:194] Registered new tserver with Master: 0ee38cb6bc634a5ca42306864554f497 (127.24.216.131:35675)
I20260929 02:12:07.836433 26016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:35553
I20260929 02:12:07.846712 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:12:07.867019 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:07.854139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37898 (local address 127.24.216.190:37005)
0929 02:12:07.854329 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:07.854334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:07.854783 (+ 449us) server_negotiation.cc:407] Connection header received
0929 02:12:07.854975 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:07.854980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:07.855049 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:07.855166 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:07.856635 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.857352 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.858731 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.859144 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.859282 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:07.861426 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:07.861432 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:07.861436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:07.861477 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:07.864436 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:07.865061 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:07.865069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:07.865072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:07.865203 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:07.865576 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:07.865582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:07.865586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:07.865905 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:07.866155 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:07.866522 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 02:12:07.866677 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:12:07.876874 26205 heartbeater.cc:503] Master 127.24.216.190:37005 was elected leader, sending a full tablet report...
I20260929 02:12:07.872890 26016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:12:07.880609 26016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:12:07.905328 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:07.892603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:35675 (local address 127.0.0.1:38936)
0929 02:12:07.894850 (+ 2247us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:07.894900 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:07.895083 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:07.896200 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:07.896211 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:07.896254 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:07.896636 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:07.896647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.898983 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.898991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:07.901698 (+ 2707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:07.901717 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.904300 (+ 2583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.904307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:07.904509 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:07.904569 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 02:12:07.904691 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 02:12:07.904885 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1360,"thread_start_us":1241,"threads_started":1}
I20260929 02:12:07.906924 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:07.893021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.24.216.131:35675)
0929 02:12:07.894664 (+ 1643us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:07.894676 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:07.895268 (+ 592us) server_negotiation.cc:407] Connection header received
0929 02:12:07.895376 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:07.895386 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:07.895664 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:07.895876 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:07.896849 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.898802 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.901923 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.903727 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.903957 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:07.904160 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:07.906256 (+ 2096us) server_negotiation.cc:299] Negotiation successful
0929 02:12:07.906471 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1440,"thread_start_us":1265,"threads_started":1}
I20260929 02:12:07.912393 26407 tablet_service.cc:1467] Processing CreateTablet for tablet a511f309cf0d4bd6adf93a0e01e19f88 (DEFAULT_TABLE table=test-table [id=224b1291f3e345908741c7c8f71fbe9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:07.913832 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a511f309cf0d4bd6adf93a0e01e19f88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:07.916168 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:07.893380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:42009 (local address 127.0.0.1:52378)
0929 02:12:07.905967 (+ 12587us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:07.905990 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:07.906097 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:07.908706 (+ 2609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:07.908714 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:07.908747 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:07.908985 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:07.908994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.911889 (+ 2895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.911894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:07.913541 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:07.913552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.915225 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.915231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:07.915437 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:07.915474 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:12:07.915551 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 02:12:07.915741 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12467,"mutex_wait_us":98,"thread_start_us":18247,"threads_started":1}
I20260929 02:12:07.917189 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:07.893962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.24.216.129:42009)
0929 02:12:07.907960 (+ 13998us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:07.907972 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:07.908009 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:12:07.908123 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:07.908135 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:07.908450 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:07.909147 (+ 697us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:07.909375 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.911345 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.913701 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.915044 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.916391 (+ 1347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:07.916572 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:07.916650 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 02:12:07.916841 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13818,"thread_start_us":13661,"threads_started":1}
I20260929 02:12:07.917328 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:07.894980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49038 (local address 127.24.216.130:45263)
0929 02:12:07.907233 (+ 12253us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:07.907245 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:07.908151 (+ 906us) server_negotiation.cc:407] Connection header received
0929 02:12:07.908436 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:07.908443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:07.908657 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:07.908834 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:07.910126 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.911612 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.914098 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.915610 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.915794 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:07.915983 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:07.916830 (+ 847us) server_negotiation.cc:299] Negotiation successful
0929 02:12:07.917003 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12034,"thread_start_us":12247,"threads_started":1}
I20260929 02:12:07.917562 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:07.894720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:45263 (local address 127.0.0.1:49038)
0929 02:12:07.908041 (+ 13321us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:07.908059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:07.908252 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:07.909102 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:07.909106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:07.909527 (+ 421us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:07.909926 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:07.909937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.911804 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.911809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:07.913898 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:07.913911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:07.916157 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:07.916167 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:07.916622 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:07.916647 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 02:12:07.916762 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 02:12:07.916892 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13187,"mutex_wait_us":35,"thread_start_us":13047,"threads_started":1}
I20260929 02:12:07.921077 26140 tablet_service.cc:1467] Processing CreateTablet for tablet a511f309cf0d4bd6adf93a0e01e19f88 (DEFAULT_TABLE table=test-table [id=224b1291f3e345908741c7c8f71fbe9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:07.921545 26274 tablet_service.cc:1467] Processing CreateTablet for tablet a511f309cf0d4bd6adf93a0e01e19f88 (DEFAULT_TABLE table=test-table [id=224b1291f3e345908741c7c8f71fbe9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:07.922389 26140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a511f309cf0d4bd6adf93a0e01e19f88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:07.922873 26274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a511f309cf0d4bd6adf93a0e01e19f88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:07.930861 26490 tablet_bootstrap.cc:492] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497: Bootstrap starting.
I20260929 02:12:07.934346 26491 tablet_bootstrap.cc:492] T a511f309cf0d4bd6adf93a0e01e19f88 P 0f1a517465344dacaafa33439fcf01a4: Bootstrap starting.
I20260929 02:12:07.935530 26490 tablet_bootstrap.cc:654] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:07.937129 26492 tablet_bootstrap.cc:492] T a511f309cf0d4bd6adf93a0e01e19f88 P 6c992f5d7369487f9747e7b4ba0bfb2b: Bootstrap starting.
I20260929 02:12:07.937197 26490 log.cc:824] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:07.938364 26491 tablet_bootstrap.cc:654] T a511f309cf0d4bd6adf93a0e01e19f88 P 0f1a517465344dacaafa33439fcf01a4: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:07.939716 26491 log.cc:824] T a511f309cf0d4bd6adf93a0e01e19f88 P 0f1a517465344dacaafa33439fcf01a4: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:07.940639 26490 tablet_bootstrap.cc:492] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497: No bootstrap required, opened a new log
I20260929 02:12:07.940740 26492 tablet_bootstrap.cc:654] T a511f309cf0d4bd6adf93a0e01e19f88 P 6c992f5d7369487f9747e7b4ba0bfb2b: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:07.940933 26490 ts_tablet_manager.cc:1397] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 02:12:07.941920 26492 log.cc:824] T a511f309cf0d4bd6adf93a0e01e19f88 P 6c992f5d7369487f9747e7b4ba0bfb2b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:07.945572 26492 tablet_bootstrap.cc:492] T a511f309cf0d4bd6adf93a0e01e19f88 P 6c992f5d7369487f9747e7b4ba0bfb2b: No bootstrap required, opened a new log
I20260929 02:12:07.945870 26492 ts_tablet_manager.cc:1397] T a511f309cf0d4bd6adf93a0e01e19f88 P 6c992f5d7369487f9747e7b4ba0bfb2b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 02:12:07.951264 26491 tablet_bootstrap.cc:492] T a511f309cf0d4bd6adf93a0e01e19f88 P 0f1a517465344dacaafa33439fcf01a4: No bootstrap required, opened a new log
I20260929 02:12:07.951555 26491 ts_tablet_manager.cc:1397] T a511f309cf0d4bd6adf93a0e01e19f88 P 0f1a517465344dacaafa33439fcf01a4: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.011s
I20260929 02:12:07.952414 26490 raft_consensus.cc:348] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee38cb6bc634a5ca42306864554f497" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35675 } } peers { permanent_uuid: "6c992f5d7369487f9747e7b4ba0bfb2b" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 45263 } } peers { permanent_uuid: "0f1a517465344dacaafa33439fcf01a4" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42009 } }
I20260929 02:12:07.952890 26490 raft_consensus.cc:374] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:07.952984 26490 raft_consensus.cc:729] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee38cb6bc634a5ca42306864554f497, State: Initialized, Role: FOLLOWER
I20260929 02:12:07.954025 26490 consensus_queue.cc:260] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee38cb6bc634a5ca42306864554f497" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35675 } } peers { permanent_uuid: "6c992f5d7369487f9747e7b4ba0bfb2b" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 45263 } } peers { permanent_uuid: "0f1a517465344dacaafa33439fcf01a4" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42009 } }
I20260929 02:12:07.955207 26472 heartbeater.cc:503] Master 127.24.216.190:37005 was elected leader, sending a full tablet report...
I20260929 02:12:07.957245 26490 ts_tablet_manager.cc:1428] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260929 02:12:07.957587 26492 raft_consensus.cc:348] T a511f309cf0d4bd6adf93a0e01e19f88 P 6c992f5d7369487f9747e7b4ba0bfb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee38cb6bc634a5ca42306864554f497" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35675 } } peers { permanent_uuid: "6c992f5d7369487f9747e7b4ba0bfb2b" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 45263 } } peers { permanent_uuid: "0f1a517465344dacaafa33439fcf01a4" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42009 } }
I20260929 02:12:07.957973 26492 raft_consensus.cc:374] T a511f309cf0d4bd6adf93a0e01e19f88 P 6c992f5d7369487f9747e7b4ba0bfb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:07.958092 26492 raft_consensus.cc:729] T a511f309cf0d4bd6adf93a0e01e19f88 P 6c992f5d7369487f9747e7b4ba0bfb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c992f5d7369487f9747e7b4ba0bfb2b, State: Initialized, Role: FOLLOWER
I20260929 02:12:07.958715 26492 consensus_queue.cc:260] T a511f309cf0d4bd6adf93a0e01e19f88 P 6c992f5d7369487f9747e7b4ba0bfb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee38cb6bc634a5ca42306864554f497" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35675 } } peers { permanent_uuid: "6c992f5d7369487f9747e7b4ba0bfb2b" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 45263 } } peers { permanent_uuid: "0f1a517465344dacaafa33439fcf01a4" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42009 } }
I20260929 02:12:07.960446 26492 ts_tablet_manager.cc:1428] T a511f309cf0d4bd6adf93a0e01e19f88 P 6c992f5d7369487f9747e7b4ba0bfb2b: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260929 02:12:07.960510 26339 heartbeater.cc:503] Master 127.24.216.190:37005 was elected leader, sending a full tablet report...
I20260929 02:12:07.961234 26491 raft_consensus.cc:348] T a511f309cf0d4bd6adf93a0e01e19f88 P 0f1a517465344dacaafa33439fcf01a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee38cb6bc634a5ca42306864554f497" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35675 } } peers { permanent_uuid: "6c992f5d7369487f9747e7b4ba0bfb2b" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 45263 } } peers { permanent_uuid: "0f1a517465344dacaafa33439fcf01a4" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42009 } }
I20260929 02:12:07.961658 26491 raft_consensus.cc:374] T a511f309cf0d4bd6adf93a0e01e19f88 P 0f1a517465344dacaafa33439fcf01a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:07.961757 26491 raft_consensus.cc:729] T a511f309cf0d4bd6adf93a0e01e19f88 P 0f1a517465344dacaafa33439fcf01a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f1a517465344dacaafa33439fcf01a4, State: Initialized, Role: FOLLOWER
I20260929 02:12:07.962558 26491 consensus_queue.cc:260] T a511f309cf0d4bd6adf93a0e01e19f88 P 0f1a517465344dacaafa33439fcf01a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee38cb6bc634a5ca42306864554f497" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35675 } } peers { permanent_uuid: "6c992f5d7369487f9747e7b4ba0bfb2b" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 45263 } } peers { permanent_uuid: "0f1a517465344dacaafa33439fcf01a4" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42009 } }
I20260929 02:12:07.964308 26491 ts_tablet_manager.cc:1428] T a511f309cf0d4bd6adf93a0e01e19f88 P 0f1a517465344dacaafa33439fcf01a4: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
W20260929 02:12:08.061321 26473 tablet.cc:2369] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:12:08.091714 26340 tablet.cc:2369] T a511f309cf0d4bd6adf93a0e01e19f88 P 6c992f5d7369487f9747e7b4ba0bfb2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:12:08.099577 26206 tablet.cc:2369] T a511f309cf0d4bd6adf93a0e01e19f88 P 0f1a517465344dacaafa33439fcf01a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:12:08.164759 26496 raft_consensus.cc:482] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:12:08.165027 26496 raft_consensus.cc:504] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee38cb6bc634a5ca42306864554f497" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35675 } } peers { permanent_uuid: "6c992f5d7369487f9747e7b4ba0bfb2b" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 45263 } } peers { permanent_uuid: "0f1a517465344dacaafa33439fcf01a4" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42009 } }
I20260929 02:12:08.167049 26496 leader_election.cc:288] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c992f5d7369487f9747e7b4ba0bfb2b (127.24.216.130:45263), 0f1a517465344dacaafa33439fcf01a4 (127.24.216.129:42009)
I20260929 02:12:08.173775 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:08.167725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:48787 (local address 127.24.216.129:42009)
0929 02:12:08.167965 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:08.167971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:08.167992 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:12:08.168066 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:08.168073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:08.168143 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:08.168218 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:08.168636 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.169597 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.171919 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.173000 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.173156 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:08.173229 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:08.173411 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 02:12:08.173546 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 02:12:08.173859 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:08.167363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:42009 (local address 127.24.216.131:48787)
0929 02:12:08.167589 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:08.167610 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:08.167758 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:08.168268 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:08.168274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:08.168295 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:08.168478 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:08.168485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.169724 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.169728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:08.171767 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:08.171783 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.173140 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.173143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:08.173263 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:08.173286 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:12:08.173358 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 02:12:08.173441 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":1}
I20260929 02:12:08.174439 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:08.167515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:48485 (local address 127.24.216.130:45263)
0929 02:12:08.167703 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:08.167709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:08.168286 (+ 577us) server_negotiation.cc:407] Connection header received
0929 02:12:08.168412 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:08.168415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:08.168469 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:08.168531 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:08.169089 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.169996 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.172242 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.173437 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.173572 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:08.173634 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:08.174093 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 02:12:08.174270 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:12:08.174991 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:08.167448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:45263 (local address 127.24.216.131:48485)
0929 02:12:08.168149 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:08.168162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:08.168290 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:08.168571 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:08.168577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:08.168602 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:08.168940 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:08.168947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.170169 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.170175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:08.172098 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:08.172107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.173637 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.173643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:08.173895 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:08.173929 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:12:08.174609 (+ 680us) client_negotiation.cc:240] Negotiation successful
0929 02:12:08.174752 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":49,"thread_start_us":421,"threads_started":1}
I20260929 02:12:08.174839 26160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a511f309cf0d4bd6adf93a0e01e19f88" candidate_uuid: "0ee38cb6bc634a5ca42306864554f497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1a517465344dacaafa33439fcf01a4" is_pre_election: true
I20260929 02:12:08.175359 26160 raft_consensus.cc:2454] T a511f309cf0d4bd6adf93a0e01e19f88 P 0f1a517465344dacaafa33439fcf01a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee38cb6bc634a5ca42306864554f497 in term 0.
I20260929 02:12:08.176041 26362 leader_election.cc:302] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee38cb6bc634a5ca42306864554f497, 0f1a517465344dacaafa33439fcf01a4; no voters:
I20260929 02:12:08.175959 26294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a511f309cf0d4bd6adf93a0e01e19f88" candidate_uuid: "0ee38cb6bc634a5ca42306864554f497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c992f5d7369487f9747e7b4ba0bfb2b" is_pre_election: true
I20260929 02:12:08.176440 26496 raft_consensus.cc:2781] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:12:08.176591 26496 raft_consensus.cc:482] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:12:08.176702 26496 raft_consensus.cc:3037] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:08.176889 26294 raft_consensus.cc:2454] T a511f309cf0d4bd6adf93a0e01e19f88 P 6c992f5d7369487f9747e7b4ba0bfb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee38cb6bc634a5ca42306864554f497 in term 0.
I20260929 02:12:08.179011 26496 raft_consensus.cc:504] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee38cb6bc634a5ca42306864554f497" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35675 } } peers { permanent_uuid: "6c992f5d7369487f9747e7b4ba0bfb2b" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 45263 } } peers { permanent_uuid: "0f1a517465344dacaafa33439fcf01a4" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42009 } }
I20260929 02:12:08.179661 26496 leader_election.cc:288] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [CANDIDATE]: Term 1 election: Requested vote from peers 6c992f5d7369487f9747e7b4ba0bfb2b (127.24.216.130:45263), 0f1a517465344dacaafa33439fcf01a4 (127.24.216.129:42009)
I20260929 02:12:08.179986 26294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a511f309cf0d4bd6adf93a0e01e19f88" candidate_uuid: "0ee38cb6bc634a5ca42306864554f497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c992f5d7369487f9747e7b4ba0bfb2b"
I20260929 02:12:08.180013 26160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a511f309cf0d4bd6adf93a0e01e19f88" candidate_uuid: "0ee38cb6bc634a5ca42306864554f497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f1a517465344dacaafa33439fcf01a4"
I20260929 02:12:08.180154 26294 raft_consensus.cc:3037] T a511f309cf0d4bd6adf93a0e01e19f88 P 6c992f5d7369487f9747e7b4ba0bfb2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:08.180190 26160 raft_consensus.cc:3037] T a511f309cf0d4bd6adf93a0e01e19f88 P 0f1a517465344dacaafa33439fcf01a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:08.182344 26294 raft_consensus.cc:2454] T a511f309cf0d4bd6adf93a0e01e19f88 P 6c992f5d7369487f9747e7b4ba0bfb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee38cb6bc634a5ca42306864554f497 in term 1.
I20260929 02:12:08.182363 26160 raft_consensus.cc:2454] T a511f309cf0d4bd6adf93a0e01e19f88 P 0f1a517465344dacaafa33439fcf01a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee38cb6bc634a5ca42306864554f497 in term 1.
I20260929 02:12:08.182720 26360 leader_election.cc:302] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee38cb6bc634a5ca42306864554f497, 6c992f5d7369487f9747e7b4ba0bfb2b; no voters:
I20260929 02:12:08.183000 26496 raft_consensus.cc:2781] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:08.183458 26496 raft_consensus.cc:686] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [term 1 LEADER]: Becoming Leader. State: Replica: 0ee38cb6bc634a5ca42306864554f497, State: Running, Role: LEADER
I20260929 02:12:08.183938 26496 consensus_queue.cc:237] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee38cb6bc634a5ca42306864554f497" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35675 } } peers { permanent_uuid: "6c992f5d7369487f9747e7b4ba0bfb2b" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 45263 } } peers { permanent_uuid: "0f1a517465344dacaafa33439fcf01a4" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42009 } }
I20260929 02:12:08.188258 26016 catalog_manager.cc:5495] T a511f309cf0d4bd6adf93a0e01e19f88 P 0ee38cb6bc634a5ca42306864554f497 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ee38cb6bc634a5ca42306864554f497 (127.24.216.131). New cstate: current_term: 1 leader_uuid: "0ee38cb6bc634a5ca42306864554f497" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee38cb6bc634a5ca42306864554f497" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c992f5d7369487f9747e7b4ba0bfb2b" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 45263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f1a517465344dacaafa33439fcf01a4" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42009 } health_report { overall_health: UNKNOWN } } }
Sep 29 02:12:08 dist-test-slave-9k0h krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:12:08.273977 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:08.258990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.24.216.129:42009)
0929 02:12:08.259222 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:08.259231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:08.259757 (+ 526us) server_negotiation.cc:407] Connection header received
0929 02:12:08.259892 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:08.259896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:08.259956 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:08.260049 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:08.261423 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.262280 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.263576 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.263989 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.264127 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:08.268022 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:08.268035 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:08.268050 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:08.268080 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:08.270908 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:08.271547 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:08.271554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:08.271556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:08.271678 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:08.272031 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:08.272037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:08.272040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:08.272395 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:08.272691 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:08.272985 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 02:12:08.273591 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 02:12:08 dist-test-slave-9k0h krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:12:08 dist-test-slave-9k0h krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:12:08.320746 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:08.307201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52394 (local address 127.24.216.129:42009)
0929 02:12:08.307461 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:08.307469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:08.307622 (+ 153us) server_negotiation.cc:407] Connection header received
0929 02:12:08.307765 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:08.307769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:08.307828 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:08.307912 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:08.309222 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.309945 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.311161 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.311541 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.311639 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:08.315854 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:08.315859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:08.315862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:08.315891 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:08.318481 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:08.318961 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:08.318965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:08.318967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:08.319100 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:08.319485 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:08.319488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:08.319490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:08.319725 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:08.319926 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:08.320126 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 02:12:08.320442 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
Sep 29 02:12:08 dist-test-slave-9k0h krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:12:08 dist-test-slave-9k0h krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:12:08.363648 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:08.351039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52396 (local address 127.24.216.129:42009)
0929 02:12:08.351248 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:08.351256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:08.351414 (+ 158us) server_negotiation.cc:407] Connection header received
0929 02:12:08.351544 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:08.351548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:08.351605 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:08.351683 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:08.352948 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.353679 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.354856 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.355239 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.355332 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:08.359349 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:08.359353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:08.359357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:08.359394 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:08.361596 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:08.362188 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:08.362195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:08.362199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:08.362313 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:08.362603 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:08.362606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:08.362608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:08.362870 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:08.363036 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:08.363252 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 02:12:08.363410 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 02:12:08.365314 26120 tablet_service.cc:2908] Rejecting scan request for tablet a511f309cf0d4bd6adf93a0e01e19f88: Uninitialized: clean time has not yet been initialized
I20260929 02:12:08.368503 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 26077
I20260929 02:12:08.380162 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:08.369303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52408 (local address 127.24.216.129:42009)
0929 02:12:08.369501 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:08.369506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:08.369719 (+ 213us) server_negotiation.cc:407] Connection header received
0929 02:12:08.369878 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:08.369881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:08.369943 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:08.370017 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:08.371124 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.371755 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.372923 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.373305 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.373401 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:08.375538 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:08.375547 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:08.375555 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:08.375579 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:08.378014 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:08.378614 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:08.378617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:08.378620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:08.378721 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:08.379015 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:08.379018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:08.379020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:08.379295 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:08.379503 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:08.379686 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 02:12:08.379836 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:12:08.380779 26200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:08.468919 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 26077
I20260929 02:12:08.485662 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 26211
Sep 29 02:12:08 dist-test-slave-9k0h krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:12:08.499527 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:08.486406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.24.216.130:45263)
0929 02:12:08.486592 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:08.486599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:08.486697 (+ 98us) server_negotiation.cc:407] Connection header received
0929 02:12:08.486927 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:08.486932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:08.487001 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:08.487107 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:08.488251 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.488960 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.490311 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.490674 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.490834 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:08.494623 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:08.494635 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:08.494652 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:08.494684 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:08.497168 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:08.497721 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:08.497728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:08.497730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:08.497857 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:08.498270 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:08.498278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:08.498283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:08.498633 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:08.498851 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:08.499104 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 02:12:08.499255 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:12:08.500144 26334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:08.584982 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 26211
I20260929 02:12:08.601419 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 26344
Sep 29 02:12:08 dist-test-slave-9k0h krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:12:08.615491 26515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:08.602098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.24.216.131:35675)
0929 02:12:08.602618 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:08.602622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:08.602639 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:12:08.602691 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:08.602694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:08.602755 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:08.602847 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:08.603989 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.604679 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.605882 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:08.606390 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:08.606483 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:08.610258 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:08.610278 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:08.610294 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:08.610322 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:08.612843 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:08.613481 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:08.613488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:08.613491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:08.613627 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:08.613967 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:08.613970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:08.613972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:08.614355 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:08.614668 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:08.614908 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 02:12:08.615111 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":322,"threads_started":1}
I20260929 02:12:08.616130 26467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:08.702565 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 26344
I20260929 02:12:08.719954 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 25984
I20260929 02:12:08.721175 26046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:08.826488 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 25984
2026-09-29T02:12:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4197 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:12:08 dist-test-slave-9k0h krb5kdc[26520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:12:08 dist-test-slave-9k0h krb5kdc[26520](info): set up 2 sockets
Sep 29 02:12:08 dist-test-slave-9k0h krb5kdc[26520](info): commencing operation
krb5kdc: starting...
W20260929 02:12:09.890041 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:12:09 dist-test-slave-9k0h krb5kdc[26520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:12:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:12:10.057132 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:35131
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:35131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:10.361492 26533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:10.361773 26533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:10.361889 26533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:10.368790 26533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:10.368882 26533 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:12:10.368981 26533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:10.369084 26533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:10.369171 26533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:10.369292 26533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:12:10.378949 26533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:35131
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:35131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:10.380210 26533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:10.381474 26533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:10.392782 26538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:10.392817 26539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:10.392856 26541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:10.393652 26533 server_base.cc:1034] running on GCE node
I20260929 02:12:10.394443 26533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:10.395663 26533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:10.396927 26533 hybrid_clock.cc:648] HybridClock initialized: now 1790647930396840 us; error 73 us; skew 500 ppm
I20260929 02:12:10.397325 26533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:10 dist-test-slave-9k0h krb5kdc[26520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647930, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:10.400343 26533 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:12:10.401626 26533 webserver.cc:460] Webserver started at http://127.24.216.190:40009/ using document root <none> and password file <none>
I20260929 02:12:10.402357 26533 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:10.402504 26533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:10.402853 26533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:10.405614 26533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "42b28fed4c3f422da79979f3fda0941e"
format_stamp: "Formatted at 2026-09-29 02:12:10 on dist-test-slave-9k0h"
server_key: "861b44f005d9ee9e5da120c2fd253c3a"
server_key_iv: "ada5e61a2f1c3ee2150f4606cc07472d"
server_key_version: "encryptionkey@0"
I20260929 02:12:10.406482 26533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "42b28fed4c3f422da79979f3fda0941e"
format_stamp: "Formatted at 2026-09-29 02:12:10 on dist-test-slave-9k0h"
server_key: "861b44f005d9ee9e5da120c2fd253c3a"
server_key_iv: "ada5e61a2f1c3ee2150f4606cc07472d"
server_key_version: "encryptionkey@0"
I20260929 02:12:10.411360 26533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 02:12:10.414388 26548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:10.415573 26533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 02:12:10.415741 26533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "42b28fed4c3f422da79979f3fda0941e"
format_stamp: "Formatted at 2026-09-29 02:12:10 on dist-test-slave-9k0h"
server_key: "861b44f005d9ee9e5da120c2fd253c3a"
server_key_iv: "ada5e61a2f1c3ee2150f4606cc07472d"
server_key_version: "encryptionkey@0"
I20260929 02:12:10.415953 26533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:10.427310 26533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:10.430794 26533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:10.431036 26533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:10.451006 26533 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:35131
I20260929 02:12:10.451025 26609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:35131 every 8 connection(s)
I20260929 02:12:10.452251 26533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:10.455848 26610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:10.462142 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 26533
I20260929 02:12:10.462405 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:10.463200 25442 external_mini_cluster.cc:1428] Setting key ac316eda2ff3c4b4778b0ae8d70f1610
I20260929 02:12:10.466449 26610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e: Bootstrap starting.
I20260929 02:12:10.469640 26610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:10.470754 26610 log.cc:824] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:10.473634 26610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e: No bootstrap required, opened a new log
Sep 29 02:12:10 dist-test-slave-9k0h krb5kdc[26520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:10.481520 26610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b28fed4c3f422da79979f3fda0941e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35131 } }
I20260929 02:12:10.481931 26610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:10.482089 26610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b28fed4c3f422da79979f3fda0941e, State: Initialized, Role: FOLLOWER
I20260929 02:12:10.482762 26610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b28fed4c3f422da79979f3fda0941e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35131 } }
I20260929 02:12:10.483019 26610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:10.483165 26610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:10.483377 26610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:10.485222 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:10.465114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.24.216.190:35131)
0929 02:12:10.465859 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:10.465869 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:10.465925 (+ 56us) server_negotiation.cc:407] Connection header received
0929 02:12:10.467635 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:10.467657 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:10.468067 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:10.468363 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:10.469961 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:10.471454 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:10.473084 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:10.473740 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:10.473943 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:10.478137 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:10.478186 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:10.478205 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:10.478248 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:10.481253 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:10.481991 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:10.481999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:10.482007 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:10.482249 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:10.482632 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:10.482635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:10.482637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:10.483150 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:10.483430 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:10.483862 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 02:12:10.484211 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":363,"threads_started":1}
I20260929 02:12:10.485941 26610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b28fed4c3f422da79979f3fda0941e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35131 } }
I20260929 02:12:10.486476 26610 leader_election.cc:302] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42b28fed4c3f422da79979f3fda0941e; no voters:
I20260929 02:12:10.487082 26610 leader_election.cc:288] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:10.487224 26615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:10.487852 26615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [term 1 LEADER]: Becoming Leader. State: Replica: 42b28fed4c3f422da79979f3fda0941e, State: Running, Role: LEADER
I20260929 02:12:10.488425 26615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b28fed4c3f422da79979f3fda0941e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35131 } }
I20260929 02:12:10.489871 26610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:10.491086 26616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42b28fed4c3f422da79979f3fda0941e. Latest consensus state: current_term: 1 leader_uuid: "42b28fed4c3f422da79979f3fda0941e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b28fed4c3f422da79979f3fda0941e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35131 } } }
I20260929 02:12:10.491457 26616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:10.492069 26617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42b28fed4c3f422da79979f3fda0941e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b28fed4c3f422da79979f3fda0941e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35131 } } }
I20260929 02:12:10.492475 26617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:10.501210 26620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:10.506937 26620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:10.515519 26620 catalog_manager.cc:1347] Generated new cluster ID: 549c18a7434e402c92a27f384e072cd8
I20260929 02:12:10.515630 26620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:10.530378 26620 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:10.531605 26620 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:10.539172 26620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e: Generated new TSK 0
I20260929 02:12:10.540108 26620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:12:10.587855 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35131
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:10.896505 26637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:10.896780 26637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:10.896965 26637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:10.904292 26637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:10.904440 26637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:10.904567 26637 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:12:10.904708 26637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:12:10.913923 26637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.216.190:35131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:10.915134 26637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:10.916414 26637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:10.929144 26642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:10.929098 26643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:10.929556 26645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:10.930068 26637 server_base.cc:1034] running on GCE node
I20260929 02:12:10.930642 26637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:10.931516 26637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:10.932742 26637 hybrid_clock.cc:648] HybridClock initialized: now 1790647930932663 us; error 60 us; skew 500 ppm
I20260929 02:12:10.932998 26637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:10 dist-test-slave-9k0h krb5kdc[26520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647930, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:10.935889 26637 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:12:10.937307 26637 webserver.cc:460] Webserver started at http://127.24.216.129:40735/ using document root <none> and password file <none>
I20260929 02:12:10.937948 26637 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:10.938064 26637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:10.938402 26637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:10.941176 26637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "1d92f787d46b45ebb9799645da9013d2"
format_stamp: "Formatted at 2026-09-29 02:12:10 on dist-test-slave-9k0h"
server_key: "0351dd498e2702bc537c3b647d4fbf70"
server_key_iv: "5edf91e15abd5d07ddf14ab3fc5739da"
server_key_version: "encryptionkey@0"
I20260929 02:12:10.941934 26637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "1d92f787d46b45ebb9799645da9013d2"
format_stamp: "Formatted at 2026-09-29 02:12:10 on dist-test-slave-9k0h"
server_key: "0351dd498e2702bc537c3b647d4fbf70"
server_key_iv: "5edf91e15abd5d07ddf14ab3fc5739da"
server_key_version: "encryptionkey@0"
I20260929 02:12:10.946749 26637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:12:10.950382 26652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:10.951609 26637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:12:10.951767 26637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "1d92f787d46b45ebb9799645da9013d2"
format_stamp: "Formatted at 2026-09-29 02:12:10 on dist-test-slave-9k0h"
server_key: "0351dd498e2702bc537c3b647d4fbf70"
server_key_iv: "5edf91e15abd5d07ddf14ab3fc5739da"
server_key_version: "encryptionkey@0"
I20260929 02:12:10.951963 26637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:10.999138 26637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:11.003471 26637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:11.003754 26637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:11.006325 26637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:11.006431 26637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:11.006584 26637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:11.006660 26637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:12:11 dist-test-slave-9k0h krb5kdc[26520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647930, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:11.028081 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:11.007040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:34191 (local address 127.24.216.190:35131)
0929 02:12:11.007629 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:11.007635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:11.008942 (+ 1307us) server_negotiation.cc:407] Connection header received
0929 02:12:11.010988 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:11.010993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:11.011099 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:11.011229 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:11.013301 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.014563 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.017143 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.017725 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.017858 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:11.022811 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:11.022816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:11.022819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:11.022849 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:11.025199 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:11.025873 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:11.025879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:11.025882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:11.026022 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:11.026438 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:11.026445 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:11.026449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:11.026698 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:11.026850 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:11.027538 (+ 688us) server_negotiation.cc:299] Negotiation successful
0929 02:12:11.027735 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":368,"threads_started":1}
I20260929 02:12:11.029263 26662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:11.007908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:35131 (local address 127.24.216.129:34191)
0929 02:12:11.008772 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:11.008817 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:11.010708 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:11.011495 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:11.011508 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:11.012297 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:11.013078 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:11.013127 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.014806 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.014813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:11.016990 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:11.017002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.017958 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.017965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:11.018138 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:11.018947 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:11.018981 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:11.022589 (+ 3608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:11.025354 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:11.025362 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:11.025377 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:11.025768 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:11.026130 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:11.026136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:11.026140 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:11.026329 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:11.026825 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:11.026833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:11.027183 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 02:12:11.027450 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 02:12:11.027931 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":355,"threads_started":1}
I20260929 02:12:11.050451 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:11.035776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:35131 (local address 127.24.216.129:58433)
0929 02:12:11.036366 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:11.036386 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:11.036508 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:11.036876 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:11.036879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:11.037267 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:11.037519 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:11.037527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.039236 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.039244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:11.040251 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:11.040264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.041422 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.041431 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:11.041652 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:11.042522 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:11.042550 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:11.043687 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:11.047896 (+ 4209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:11.047899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:11.047902 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:11.048275 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:11.048745 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:11.048748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:11.048750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:11.048871 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:11.049528 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:11.049531 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:11.049694 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 02:12:11.049829 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 02:12:11.050062 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":391,"threads_started":1}
I20260929 02:12:11.050716 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:11.036101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:58433 (local address 127.24.216.190:35131)
0929 02:12:11.036332 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:11.036340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:11.036468 (+ 128us) server_negotiation.cc:407] Connection header received
0929 02:12:11.036617 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:11.036625 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:11.036723 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:11.036845 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:11.037645 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.039087 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.040384 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.041211 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.041365 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:11.044024 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:11.044032 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:11.044040 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:11.044079 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:11.047751 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:11.048405 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:11.048424 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:11.048430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:11.048620 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:11.048973 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:11.048980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:11.048986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:11.049401 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:11.049573 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:11.049978 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 02:12:11.050282 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:12:11.064566 26637 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:41669
I20260929 02:12:11.064589 26772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:41669 every 8 connection(s)
I20260929 02:12:11.065739 26637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:12:11.067708 26773 heartbeater.cc:347] Connected to a master server at 127.24.216.190:35131
I20260929 02:12:11.067896 26773 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:11.068506 26773 heartbeater.cc:511] Master 127.24.216.190:35131 requested a full tablet report, sending...
I20260929 02:12:11.070478 26565 ts_manager.cc:194] Registered new tserver with Master: 1d92f787d46b45ebb9799645da9013d2 (127.24.216.129:41669)
I20260929 02:12:11.071882 26565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:34191
I20260929 02:12:11.073381 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 26637
I20260929 02:12:11.073596 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:12:11.074443 25442 external_mini_cluster.cc:1428] Setting key 297bf763a40d28967956114e5765955a
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:12:11.114532 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35131
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:11.437033 26777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:11.437294 26777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:11.437446 26777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:11.444360 26777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:11.444455 26777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:11.444530 26777 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:12:11.444684 26777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:12:11.453750 26777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.216.190:35131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:11.454975 26777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:11.456260 26777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:11.469316 26782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:11.469342 26783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:11.469375 26785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:11.470041 26777 server_base.cc:1034] running on GCE node
I20260929 02:12:11.470642 26777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:11.471517 26777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:11.472718 26777 hybrid_clock.cc:648] HybridClock initialized: now 1790647931472666 us; error 26 us; skew 500 ppm
I20260929 02:12:11.473042 26777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:11 dist-test-slave-9k0h krb5kdc[26520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647931, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:11.481849 26777 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:12:11.483358 26777 webserver.cc:460] Webserver started at http://127.24.216.130:42871/ using document root <none> and password file <none>
I20260929 02:12:11.484086 26777 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:11.484216 26777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:11.484526 26777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:11.487357 26777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "1a9ed8f389a5447388269d6bb541515e"
format_stamp: "Formatted at 2026-09-29 02:12:11 on dist-test-slave-9k0h"
server_key: "5ac85b87b684029b859f88d8e3ed17bb"
server_key_iv: "dae9658ac656546ada5dfa8778333b0b"
server_key_version: "encryptionkey@0"
I20260929 02:12:11.488103 26777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "1a9ed8f389a5447388269d6bb541515e"
format_stamp: "Formatted at 2026-09-29 02:12:11 on dist-test-slave-9k0h"
server_key: "5ac85b87b684029b859f88d8e3ed17bb"
server_key_iv: "dae9658ac656546ada5dfa8778333b0b"
server_key_version: "encryptionkey@0"
I20260929 02:12:11.492853 26777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:12:11.496042 26792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:11.497272 26777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:11.497433 26777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "1a9ed8f389a5447388269d6bb541515e"
format_stamp: "Formatted at 2026-09-29 02:12:11 on dist-test-slave-9k0h"
server_key: "5ac85b87b684029b859f88d8e3ed17bb"
server_key_iv: "dae9658ac656546ada5dfa8778333b0b"
server_key_version: "encryptionkey@0"
I20260929 02:12:11.497637 26777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:11.517459 26777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:11.521310 26777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:11.521571 26777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:11.524140 26777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:11.524243 26777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:11.524385 26777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:11.524514 26777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 29 02:12:11 dist-test-slave-9k0h krb5kdc[26520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647931, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:11.545863 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:11.524610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:41241 (local address 127.24.216.190:35131)
0929 02:12:11.524818 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:11.524823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:11.526126 (+ 1303us) server_negotiation.cc:407] Connection header received
0929 02:12:11.528091 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:11.528096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:11.528180 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:11.528290 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:11.530388 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.531359 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.534619 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.535049 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.535144 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:11.540232 (+ 5088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:11.540240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:11.540250 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:11.540286 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:11.543020 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:11.543679 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:11.543684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:11.543688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:11.543819 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:11.544249 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:11.544254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:11.544257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:11.544508 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:11.544665 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:11.545290 (+ 625us) server_negotiation.cc:299] Negotiation successful
0929 02:12:11.545495 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:12:11.547250 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:11.525210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:35131 (local address 127.24.216.130:41241)
0929 02:12:11.525954 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:11.526010 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:11.527798 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:11.528552 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:11.528570 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:11.529290 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:11.530126 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:11.530169 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.531511 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.531518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:11.534456 (+ 2938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:11.534473 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.535329 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.535339 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:11.535688 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:11.536429 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:11.536471 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:11.540001 (+ 3530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:11.543145 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:11.543153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:11.543167 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:11.543555 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:11.543945 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:11.543950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:11.543953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:11.544135 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:11.544632 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:11.544639 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:11.544959 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 02:12:11.545204 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 02:12:11.545696 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":313,"threads_started":1}
I20260929 02:12:11.568059 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:11.554131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:55647 (local address 127.24.216.190:35131)
0929 02:12:11.554495 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:11.554503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:11.554816 (+ 313us) server_negotiation.cc:407] Connection header received
0929 02:12:11.555012 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:11.555017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:11.555087 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:11.555161 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:11.556163 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.556916 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.558407 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.558989 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.559132 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:11.561325 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:11.561330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:11.561333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:11.561359 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:11.564852 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:11.565719 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:11.565724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:11.565728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:11.565892 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:11.566395 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:11.566404 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:11.566409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:11.566886 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:11.567054 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:11.567519 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 02:12:11.567738 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260929 02:12:11.568166 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:11.554126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:35131 (local address 127.24.216.130:55647)
0929 02:12:11.554699 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:11.554717 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:11.554872 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:11.555259 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:11.555264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:11.555678 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:11.556014 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:11.556024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.557045 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.557050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:11.558254 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:11.558276 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:11.559170 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:11.559177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:11.559378 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:11.559996 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:11.560011 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:11.561067 (+ 1056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:11.564980 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:11.564988 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:11.564997 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:11.565535 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:11.566022 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:11.566027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:11.566031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:11.566265 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:11.567027 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:11.567034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:11.567238 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 02:12:11.567391 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 02:12:11.567744 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":363,"threads_started":1}
I20260929 02:12:11.578644 26777 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:38709
I20260929 02:12:11.578672 26911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:38709 every 8 connection(s)
I20260929 02:12:11.579811 26777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:12:11.581866 26912 heartbeater.cc:347] Connected to a master server at 127.24.216.190:35131
I20260929 02:12:11.582044 26912 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:11.582660 26912 heartbeater.cc:511] Master 127.24.216.190:35131 requested a full tablet report, sending...
I20260929 02:12:11.583914 26565 ts_manager.cc:194] Registered new tserver with Master: 1a9ed8f389a5447388269d6bb541515e (127.24.216.130:38709)
I20260929 02:12:11.584686 26565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:41241
I20260929 02:12:11.584789 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 26777
I20260929 02:12:11.585016 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:12:11.585453 25442 external_mini_cluster.cc:1428] Setting key 70e271ad9cae28b1afb5a2f2c9c73d91
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:12:11.625488 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35131
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:11.931821 26916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:11.932111 26916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:11.932247 26916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:11.939018 26916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:11.939105 26916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:11.939188 26916 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:12:11.939268 26916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:12:11.948542 26916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.216.190:35131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:11.949843 26916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:11.951184 26916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:11.963850 26921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:11.963871 26924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:11.963816 26922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:11.964877 26916 server_base.cc:1034] running on GCE node
I20260929 02:12:11.965408 26916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:11.966305 26916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:11.967517 26916 hybrid_clock.cc:648] HybridClock initialized: now 1790647931967452 us; error 50 us; skew 500 ppm
I20260929 02:12:11.967796 26916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:11 dist-test-slave-9k0h krb5kdc[26520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647931, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:11.970692 26916 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:12:11.971930 26916 webserver.cc:460] Webserver started at http://127.24.216.131:32877/ using document root <none> and password file <none>
I20260929 02:12:11.972561 26916 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:11.972643 26916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:11.972899 26916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:11.975864 26916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "a8504c9357884d9483789b7c9361ea68"
format_stamp: "Formatted at 2026-09-29 02:12:11 on dist-test-slave-9k0h"
server_key: "9b34157a111bf138bbb23c956d3ab541"
server_key_iv: "bf66985acba4ebb7f04633b5fa5acea8"
server_key_version: "encryptionkey@0"
I20260929 02:12:11.976554 26916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "a8504c9357884d9483789b7c9361ea68"
format_stamp: "Formatted at 2026-09-29 02:12:11 on dist-test-slave-9k0h"
server_key: "9b34157a111bf138bbb23c956d3ab541"
server_key_iv: "bf66985acba4ebb7f04633b5fa5acea8"
server_key_version: "encryptionkey@0"
I20260929 02:12:11.981350 26916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 02:12:11.984606 26931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:11.985826 26916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:12:11.986009 26916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "a8504c9357884d9483789b7c9361ea68"
format_stamp: "Formatted at 2026-09-29 02:12:11 on dist-test-slave-9k0h"
server_key: "9b34157a111bf138bbb23c956d3ab541"
server_key_iv: "bf66985acba4ebb7f04633b5fa5acea8"
server_key_version: "encryptionkey@0"
I20260929 02:12:11.986284 26916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:12.000919 26916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:12.005314 26916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:12.005569 26916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:12.007977 26916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:12.008087 26916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:12.008231 26916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:12.008301 26916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:12:12 dist-test-slave-9k0h krb5kdc[26520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647931, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:12.032939 26940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.009505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:35131 (local address 127.24.216.131:34731)
0929 02:12:12.010406 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:12.010442 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:12.012233 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:12.012921 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:12.012929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:12.013612 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:12.014551 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.014583 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.015930 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.015938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.018038 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.018070 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.019200 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.019208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.019469 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.020236 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:12.020268 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:12.023403 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:12.027750 (+ 4347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:12.027761 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:12.027778 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:12.028431 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:12.028914 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:12.028917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:12.028919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:12.029168 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:12.029857 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:12.029864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:12.030431 (+ 567us) client_negotiation.cc:769] Sending connection context
0929 02:12:12.030774 (+ 343us) client_negotiation.cc:240] Negotiation successful
0929 02:12:12.031319 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":434,"threads_started":1}
I20260929 02:12:12.033519 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.008882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:34731 (local address 127.24.216.190:35131)
0929 02:12:12.009071 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:12.009079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:12.010569 (+ 1490us) server_negotiation.cc:407] Connection header received
0929 02:12:12.012491 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:12.012496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:12.012576 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:12.012692 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:12.014715 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.015781 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.018254 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.018931 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.019103 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.023679 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:12.023686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:12.023693 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:12.023732 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:12.027602 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:12.028598 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:12.028606 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:12.028612 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:12.028791 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:12.029297 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:12.029316 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:12.029319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:12.029705 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:12.029861 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:12.033002 (+ 3141us) server_negotiation.cc:299] Negotiation successful
0929 02:12:12.033232 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:12:12.052690 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.038709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:35131 (local address 127.24.216.131:55467)
0929 02:12:12.039414 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:12.039429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:12.039527 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:12.039857 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:12.039860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:12.040242 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:12.040498 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.040505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.042051 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.042060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.043730 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.043741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.044473 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.044478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.044639 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.045287 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:12.045302 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:12.045940 (+ 638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:12.049922 (+ 3982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:12.049925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:12.049928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:12.050394 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:12.050824 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:12.050832 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:12.050838 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:12.051002 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:12.051695 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:12.051699 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:12.051936 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 02:12:12.052115 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 02:12:12.052390 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":390,"threads_started":1}
I20260929 02:12:12.053002 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.038685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:55467 (local address 127.24.216.190:35131)
0929 02:12:12.038893 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:12.038902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:12.039490 (+ 588us) server_negotiation.cc:407] Connection header received
0929 02:12:12.039616 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:12.039624 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:12.039718 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:12.039840 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:12.040626 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.041907 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.043886 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.044291 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.044435 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.046123 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:12.046129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:12.046134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:12.046208 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:12.049811 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:12.050503 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:12.050508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:12.050511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:12.050672 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:12.051114 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:12.051122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:12.051125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:12.051489 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:12.051638 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:12.052221 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 02:12:12.052515 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:12:12.060176 26916 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:38255
I20260929 02:12:12.060199 27050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:38255 every 8 connection(s)
I20260929 02:12:12.061370 26916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:12:12.061609 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 26916
I20260929 02:12:12.061822 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:12:12.062386 25442 external_mini_cluster.cc:1428] Setting key b11e3f503b31db12919816bf47109f6b
I20260929 02:12:12.063400 27051 heartbeater.cc:347] Connected to a master server at 127.24.216.190:35131
I20260929 02:12:12.063602 27051 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:12.064369 27051 heartbeater.cc:511] Master 127.24.216.190:35131 requested a full tablet report, sending...
I20260929 02:12:12.065734 26565 ts_manager.cc:194] Registered new tserver with Master: a8504c9357884d9483789b7c9361ea68 (127.24.216.131:38255)
I20260929 02:12:12.066787 26565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:34731
I20260929 02:12:12.075073 26773 heartbeater.cc:503] Master 127.24.216.190:35131 was elected leader, sending a full tablet report...
I20260929 02:12:12.077392 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:12:12 dist-test-slave-9k0h krb5kdc[26520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:12:12 dist-test-slave-9k0h krb5kdc[26520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:12.121080 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.108522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.24.216.190:35131)
0929 02:12:12.108754 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:12.108762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:12.108873 (+ 111us) server_negotiation.cc:407] Connection header received
0929 02:12:12.109013 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:12.109017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:12.109077 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:12.109157 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:12.110508 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.111245 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.112406 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.112790 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.112885 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.116358 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:12.116363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:12.116366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:12.116395 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:12.118947 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:12.119490 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:12.119494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:12.119496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:12.119627 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:12.119942 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:12.119945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:12.119947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:12.120188 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:12.120424 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:12.120578 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 02:12:12.120724 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 02:12:12.124876 26565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:12:12.131570 26565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:12:12.153384 27064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.142739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:38255 (local address 127.0.0.1:37092)
0929 02:12:12.143796 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:12.143851 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:12.144011 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:12.146134 (+ 2123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:12.146141 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:12.146190 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:12.146630 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.146644 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.148446 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.148451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.150692 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.150706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.152390 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.152396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.152570 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.152618 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 02:12:12.152796 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 02:12:12.152968 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":1169,"threads_started":1}
I20260929 02:12:12.153532 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.143074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.24.216.131:38255)
0929 02:12:12.145459 (+ 2385us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:12.145470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:12.145510 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:12:12.145631 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:12.145641 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:12.145925 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:12.146134 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:12.146856 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.148323 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.150847 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.152177 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.152405 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.152577 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:12.153014 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 02:12:12.153175 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2181,"thread_start_us":1995,"threads_started":1}
I20260929 02:12:12.154327 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.145204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.24.216.129:41669)
0929 02:12:12.145717 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:12.145726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:12.146267 (+ 541us) server_negotiation.cc:407] Connection header received
0929 02:12:12.146562 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:12.146570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:12.146819 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:12.146980 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:12.147620 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.149021 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.151374 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.152570 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.152746 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.152899 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:12.153801 (+ 902us) server_negotiation.cc:299] Negotiation successful
0929 02:12:12.153977 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":261,"threads_started":1}
I20260929 02:12:12.154407 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.143471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:38709 (local address 127.0.0.1:50618)
0929 02:12:12.145767 (+ 2296us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:12.145791 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:12.146544 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:12.147403 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:12.147408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:12.147437 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:12.147807 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.147817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.149522 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.149529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.151733 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.151749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.153643 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.153649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.153820 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.153848 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:12:12.153923 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 02:12:12.154035 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1402,"thread_start_us":1270,"threads_started":1}
I20260929 02:12:12.153520 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.144863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:41669 (local address 127.0.0.1:40790)
0929 02:12:12.146136 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:12.146183 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:12.146370 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:12.147155 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:12.147158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:12.147175 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:12.147462 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.147469 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.149151 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.149156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.151220 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.151242 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.152776 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.152785 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.152929 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.152957 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:12:12.153034 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 02:12:12.153151 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1155,"mutex_wait_us":22,"thread_start_us":987,"threads_started":1}
I20260929 02:12:12.155100 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.144006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.24.216.130:38709)
0929 02:12:12.144560 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:12.144572 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:12.146438 (+ 1866us) server_negotiation.cc:407] Connection header received
0929 02:12:12.146749 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:12.146756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:12.146998 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:12.147152 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:12.147976 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.149351 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.151900 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.153198 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.153357 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.153518 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:12.154644 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0929 02:12:12.154868 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":294,"threads_started":1}
I20260929 02:12:12.157857 26981 tablet_service.cc:1467] Processing CreateTablet for tablet 5d454c436162401eade51730f6887e38 (DEFAULT_TABLE table=test-table [id=d27c0157693f49fd86afa4f7b730dbf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:12.158478 26842 tablet_service.cc:1467] Processing CreateTablet for tablet 5d454c436162401eade51730f6887e38 (DEFAULT_TABLE table=test-table [id=d27c0157693f49fd86afa4f7b730dbf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:12.158478 26703 tablet_service.cc:1467] Processing CreateTablet for tablet 5d454c436162401eade51730f6887e38 (DEFAULT_TABLE table=test-table [id=d27c0157693f49fd86afa4f7b730dbf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:12.159523 26981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d454c436162401eade51730f6887e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:12.159732 26703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d454c436162401eade51730f6887e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:12.159745 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d454c436162401eade51730f6887e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:12.170490 27070 tablet_bootstrap.cc:492] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68: Bootstrap starting.
I20260929 02:12:12.173867 27072 tablet_bootstrap.cc:492] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e: Bootstrap starting.
I20260929 02:12:12.174674 27070 tablet_bootstrap.cc:654] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:12.176100 27070 log.cc:824] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:12.176908 27071 tablet_bootstrap.cc:492] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2: Bootstrap starting.
I20260929 02:12:12.177314 27072 tablet_bootstrap.cc:654] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:12.178718 27072 log.cc:824] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:12.180323 27071 tablet_bootstrap.cc:654] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:12.180672 27070 tablet_bootstrap.cc:492] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68: No bootstrap required, opened a new log
I20260929 02:12:12.180943 27070 ts_tablet_manager.cc:1397] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260929 02:12:12.181423 27071 log.cc:824] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:12.182197 27072 tablet_bootstrap.cc:492] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e: No bootstrap required, opened a new log
I20260929 02:12:12.182461 27072 ts_tablet_manager.cc:1397] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:12:12.183673 27071 tablet_bootstrap.cc:492] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2: No bootstrap required, opened a new log
I20260929 02:12:12.183928 27071 ts_tablet_manager.cc:1397] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 02:12:12.190934 27071 raft_consensus.cc:348] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } }
I20260929 02:12:12.191339 27071 raft_consensus.cc:374] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:12.191426 27071 raft_consensus.cc:729] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d92f787d46b45ebb9799645da9013d2, State: Initialized, Role: FOLLOWER
I20260929 02:12:12.192109 27071 consensus_queue.cc:260] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } }
I20260929 02:12:12.192713 27070 raft_consensus.cc:348] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } }
I20260929 02:12:12.193111 27070 raft_consensus.cc:374] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:12.193217 27070 raft_consensus.cc:729] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8504c9357884d9483789b7c9361ea68, State: Initialized, Role: FOLLOWER
I20260929 02:12:12.193519 27072 raft_consensus.cc:348] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } }
I20260929 02:12:12.193881 27072 raft_consensus.cc:374] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:12.193761 27070 consensus_queue.cc:260] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } }
I20260929 02:12:12.193987 27072 raft_consensus.cc:729] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9ed8f389a5447388269d6bb541515e, State: Initialized, Role: FOLLOWER
I20260929 02:12:12.194595 27072 consensus_queue.cc:260] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } }
I20260929 02:12:12.195868 26912 heartbeater.cc:503] Master 127.24.216.190:35131 was elected leader, sending a full tablet report...
I20260929 02:12:12.196642 27072 ts_tablet_manager.cc:1428] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 02:12:12.198889 27051 heartbeater.cc:503] Master 127.24.216.190:35131 was elected leader, sending a full tablet report...
I20260929 02:12:12.199520 27070 ts_tablet_manager.cc:1428] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68: Time spent starting tablet: real 0.018s user 0.008s sys 0.011s
I20260929 02:12:12.199783 27071 ts_tablet_manager.cc:1428] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
W20260929 02:12:12.313195 27052 tablet.cc:2369] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:12:12.318089 26774 tablet.cc:2369] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:12:12.331775 26913 tablet.cc:2369] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:12:12.497165 27078 raft_consensus.cc:482] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:12:12.497449 27078 raft_consensus.cc:504] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } }
I20260929 02:12:12.499223 27078 leader_election.cc:288] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8504c9357884d9483789b7c9361ea68 (127.24.216.131:38255), 1d92f787d46b45ebb9799645da9013d2 (127.24.216.129:41669)
I20260929 02:12:12.506753 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.499696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:42673 (local address 127.24.216.131:38255)
0929 02:12:12.499865 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:12.499869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:12.500311 (+ 442us) server_negotiation.cc:407] Connection header received
0929 02:12:12.500514 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:12.500520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:12.500601 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:12.500762 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:12.501586 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.502510 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.504840 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.505820 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.505995 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.506068 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:12.506366 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 02:12:12.506484 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:12:12.506870 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.499535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:38255 (local address 127.24.216.130:42673)
0929 02:12:12.500209 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:12.500228 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:12.500366 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:12.500812 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:12.500818 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:12.500846 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:12.501443 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.501453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.502708 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.502713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.504699 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.504709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.506022 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.506027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.506196 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.506219 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:12:12.506341 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 02:12:12.506461 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":338,"threads_started":1}
I20260929 02:12:12.507759 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.500449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:48263 (local address 127.24.216.129:41669)
0929 02:12:12.500632 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:12.500640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:12.501635 (+ 995us) server_negotiation.cc:407] Connection header received
0929 02:12:12.501805 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:12.501810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:12.501867 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:12.501955 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:12.502524 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.503651 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.505779 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.506773 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.506954 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.507020 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:12.507520 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 02:12:12.507609 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:12:12.507947 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.500288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:41669 (local address 127.24.216.130:48263)
0929 02:12:12.501538 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:12.501551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:12.501684 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:12.501970 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:12.501974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:12.501993 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:12.502353 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.502362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.503814 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.503817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.505632 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.505643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.507152 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.507158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.507355 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.507387 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:12:12.507488 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 02:12:12.507602 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"thread_start_us":925,"threads_started":1}
I20260929 02:12:12.507794 27006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d454c436162401eade51730f6887e38" candidate_uuid: "1a9ed8f389a5447388269d6bb541515e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8504c9357884d9483789b7c9361ea68" is_pre_election: true
I20260929 02:12:12.508322 27006 raft_consensus.cc:2454] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9ed8f389a5447388269d6bb541515e in term 0.
I20260929 02:12:12.508912 26796 leader_election.cc:302] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9ed8f389a5447388269d6bb541515e, a8504c9357884d9483789b7c9361ea68; no voters:
I20260929 02:12:12.508884 26728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d454c436162401eade51730f6887e38" candidate_uuid: "1a9ed8f389a5447388269d6bb541515e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d92f787d46b45ebb9799645da9013d2" is_pre_election: true
I20260929 02:12:12.509284 27078 raft_consensus.cc:2781] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:12:12.509306 26728 raft_consensus.cc:2454] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9ed8f389a5447388269d6bb541515e in term 0.
I20260929 02:12:12.509405 27078 raft_consensus.cc:482] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:12:12.509513 27078 raft_consensus.cc:3037] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:12.511772 27078 raft_consensus.cc:504] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } }
I20260929 02:12:12.512364 27078 leader_election.cc:288] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [CANDIDATE]: Term 1 election: Requested vote from peers a8504c9357884d9483789b7c9361ea68 (127.24.216.131:38255), 1d92f787d46b45ebb9799645da9013d2 (127.24.216.129:41669)
I20260929 02:12:12.512732 27006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d454c436162401eade51730f6887e38" candidate_uuid: "1a9ed8f389a5447388269d6bb541515e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8504c9357884d9483789b7c9361ea68"
I20260929 02:12:12.512823 26728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d454c436162401eade51730f6887e38" candidate_uuid: "1a9ed8f389a5447388269d6bb541515e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d92f787d46b45ebb9799645da9013d2"
I20260929 02:12:12.512921 27006 raft_consensus.cc:3037] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:12.513037 26728 raft_consensus.cc:3037] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:12.515090 27006 raft_consensus.cc:2454] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9ed8f389a5447388269d6bb541515e in term 1.
I20260929 02:12:12.515410 26728 raft_consensus.cc:2454] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9ed8f389a5447388269d6bb541515e in term 1.
I20260929 02:12:12.515558 26796 leader_election.cc:302] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9ed8f389a5447388269d6bb541515e, a8504c9357884d9483789b7c9361ea68; no voters:
I20260929 02:12:12.515825 27078 raft_consensus.cc:2781] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:12.516232 27078 raft_consensus.cc:686] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [term 1 LEADER]: Becoming Leader. State: Replica: 1a9ed8f389a5447388269d6bb541515e, State: Running, Role: LEADER
I20260929 02:12:12.516693 27078 consensus_queue.cc:237] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } }
I20260929 02:12:12.521803 26565 catalog_manager.cc:5495] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9ed8f389a5447388269d6bb541515e (127.24.216.130). New cstate: current_term: 1 leader_uuid: "1a9ed8f389a5447388269d6bb541515e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } health_report { overall_health: HEALTHY } } }
I20260929 02:12:12.610155 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.605247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.24.216.130:38709)
0929 02:12:12.605476 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:12.605482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:12.605503 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:12:12.605713 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:12.605717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:12.605817 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:12.605952 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:12:12.606295 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.607049 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.608299 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.608665 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.608777 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.609196 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:12:12.609642 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:12:12.609749 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:12.609840 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 02:12:12.609897 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:12:12.617882 26728 raft_consensus.cc:1261] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2 [term 1 FOLLOWER]: Refusing update from remote peer 1a9ed8f389a5447388269d6bb541515e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:12.617923 27006 raft_consensus.cc:1261] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68 [term 1 FOLLOWER]: Refusing update from remote peer 1a9ed8f389a5447388269d6bb541515e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:12.618790 27081 consensus_queue.cc:1035] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:12.619170 27078 consensus_queue.cc:1035] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:12.629395 27084 mvcc.cc:205] Tried to move back new op lower bound from 7334493931994320896 to 7334493931592085504. Current Snapshot: MvccSnapshot[applied={T|T < 7334493931994320896}]
I20260929 02:12:12.632867 27085 mvcc.cc:205] Tried to move back new op lower bound from 7334493931994320896 to 7334493931592085504. Current Snapshot: MvccSnapshot[applied={T|T < 7334493931994320896}]
I20260929 02:12:12.634441 27086 mvcc.cc:205] Tried to move back new op lower bound from 7334493931994320896 to 7334493931592085504. Current Snapshot: MvccSnapshot[applied={T|T < 7334493931994320896 or (T in {7334493931994320896})}]
I20260929 02:12:12.649369 26565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:12.649638 26565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:12.653107 26565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e: Sending DeleteTablet for 3 replicas of tablet 5d454c436162401eade51730f6887e38
I20260929 02:12:12.654228 26981 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d454c436162401eade51730f6887e38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:12 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:37092
I20260929 02:12:12.654280 26842 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d454c436162401eade51730f6887e38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:12 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:50618
I20260929 02:12:12.654273 26703 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d454c436162401eade51730f6887e38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:12 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:40790
I20260929 02:12:12.657967 26565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:12:12.659148 26565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:12:12.668462 26980 tablet_service.cc:1467] Processing CreateTablet for tablet 98c9818d1a074d488de2645f9a768b9e (DEFAULT_TABLE table=test-table [id=2e19d64e8e8847549d5fd28148aa5eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:12.669044 26980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c9818d1a074d488de2645f9a768b9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:12.671084 26702 tablet_service.cc:1467] Processing CreateTablet for tablet 98c9818d1a074d488de2645f9a768b9e (DEFAULT_TABLE table=test-table [id=2e19d64e8e8847549d5fd28148aa5eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:12.672184 26702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c9818d1a074d488de2645f9a768b9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:12.674793 27096 tablet_replica.cc:330] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2: stopping tablet replica
I20260929 02:12:12.675179 26841 tablet_service.cc:1467] Processing CreateTablet for tablet 98c9818d1a074d488de2645f9a768b9e (DEFAULT_TABLE table=test-table [id=2e19d64e8e8847549d5fd28148aa5eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:12.675846 26841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c9818d1a074d488de2645f9a768b9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:12.675257 27096 raft_consensus.cc:2229] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:12.676731 27096 raft_consensus.cc:2258] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:12.679298 27071 tablet_bootstrap.cc:492] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2: Bootstrap starting.
I20260929 02:12:12.679981 27094 tablet_replica.cc:330] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68: stopping tablet replica
I20260929 02:12:12.680423 27094 raft_consensus.cc:2229] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:12.680925 27094 raft_consensus.cc:2258] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:12.681125 27071 tablet_bootstrap.cc:654] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:12.682864 27094 ts_tablet_manager.cc:1905] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:12.687023 27070 tablet_bootstrap.cc:492] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68: Bootstrap starting.
I20260929 02:12:12.687898 27072 tablet_bootstrap.cc:492] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e: Bootstrap starting.
I20260929 02:12:12.688577 27096 ts_tablet_manager.cc:1905] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:12.688840 27070 tablet_bootstrap.cc:654] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:12.689554 27072 tablet_bootstrap.cc:654] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:12.699757 27096 ts_tablet_manager.cc:1918] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:12:12.699911 27072 tablet_bootstrap.cc:492] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e: No bootstrap required, opened a new log
I20260929 02:12:12.699930 27070 tablet_bootstrap.cc:492] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68: No bootstrap required, opened a new log
I20260929 02:12:12.699961 27096 log.cc:1196] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/5d454c436162401eade51730f6887e38
I20260929 02:12:12.700064 27070 ts_tablet_manager.cc:1397] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260929 02:12:12.700075 27072 ts_tablet_manager.cc:1397] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260929 02:12:12.700580 27096 ts_tablet_manager.cc:1939] T 5d454c436162401eade51730f6887e38 P 1d92f787d46b45ebb9799645da9013d2: Deleting consensus metadata
I20260929 02:12:12.699821 27071 tablet_bootstrap.cc:492] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2: No bootstrap required, opened a new log
I20260929 02:12:12.700976 27070 raft_consensus.cc:348] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.701197 27070 raft_consensus.cc:374] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:12.701205 27071 ts_tablet_manager.cc:1397] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.008s
I20260929 02:12:12.701076 27072 raft_consensus.cc:348] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.701293 27070 raft_consensus.cc:729] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8504c9357884d9483789b7c9361ea68, State: Initialized, Role: FOLLOWER
I20260929 02:12:12.701339 27072 raft_consensus.cc:374] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:12.701450 27072 raft_consensus.cc:729] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9ed8f389a5447388269d6bb541515e, State: Initialized, Role: FOLLOWER
I20260929 02:12:12.701637 27070 consensus_queue.cc:260] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.701701 27072 consensus_queue.cc:260] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.702046 27071 raft_consensus.cc:348] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.702354 27072 ts_tablet_manager.cc:1428] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:12.702368 27071 raft_consensus.cc:374] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:12.702463 27071 raft_consensus.cc:729] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d92f787d46b45ebb9799645da9013d2, State: Initialized, Role: FOLLOWER
I20260929 02:12:12.702522 27095 tablet_replica.cc:330] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e: stopping tablet replica
I20260929 02:12:12.702728 27071 consensus_queue.cc:260] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.703435 27094 ts_tablet_manager.cc:1918] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:12:12.703552 27071 ts_tablet_manager.cc:1428] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:12.705039 27094 log.cc:1196] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/5d454c436162401eade51730f6887e38
I20260929 02:12:12.705384 27095 raft_consensus.cc:2229] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:12:12.705668 26552 catalog_manager.cc:4841] TS 1d92f787d46b45ebb9799645da9013d2 (127.24.216.129:41669): tablet 5d454c436162401eade51730f6887e38 (table test-table [id=d27c0157693f49fd86afa4f7b730dbf9]) successfully deleted
I20260929 02:12:12.705868 27094 ts_tablet_manager.cc:1939] T 5d454c436162401eade51730f6887e38 P a8504c9357884d9483789b7c9361ea68: Deleting consensus metadata
I20260929 02:12:12.706103 27095 raft_consensus.cc:2258] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:12.707736 26552 catalog_manager.cc:4841] TS a8504c9357884d9483789b7c9361ea68 (127.24.216.131:38255): tablet 5d454c436162401eade51730f6887e38 (table test-table [id=d27c0157693f49fd86afa4f7b730dbf9]) successfully deleted
I20260929 02:12:12.708453 27070 ts_tablet_manager.cc:1428] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20260929 02:12:12.709190 27095 ts_tablet_manager.cc:1905] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:12.714399 27095 ts_tablet_manager.cc:1918] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:12:12.714560 27095 log.cc:1196] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/5d454c436162401eade51730f6887e38
I20260929 02:12:12.715034 27095 ts_tablet_manager.cc:1939] T 5d454c436162401eade51730f6887e38 P 1a9ed8f389a5447388269d6bb541515e: Deleting consensus metadata
I20260929 02:12:12.716341 26549 catalog_manager.cc:4841] TS 1a9ed8f389a5447388269d6bb541515e (127.24.216.130:38709): tablet 5d454c436162401eade51730f6887e38 (table test-table [id=d27c0157693f49fd86afa4f7b730dbf9]) successfully deleted
I20260929 02:12:12.773033 27076 raft_consensus.cc:482] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:12:12.773283 27076 raft_consensus.cc:504] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.774896 27076 leader_election.cc:288] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9ed8f389a5447388269d6bb541515e (127.24.216.130:38709), a8504c9357884d9483789b7c9361ea68 (127.24.216.131:38255)
I20260929 02:12:12.781723 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.775679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:41707 (local address 127.24.216.130:38709)
0929 02:12:12.775875 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:12.775883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:12.776221 (+ 338us) server_negotiation.cc:407] Connection header received
0929 02:12:12.776543 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:12.776548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:12.776628 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:12.776677 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:12.777271 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.778364 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.779998 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.780779 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.780950 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.781024 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:12.781298 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 02:12:12.781413 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:12:12.782047 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.775273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:38709 (local address 127.24.216.129:41707)
0929 02:12:12.776025 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:12.776048 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:12.776358 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:12.776836 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:12.776839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:12.776868 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:12.777120 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.777127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.778494 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.778500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.779847 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.779859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.780980 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.780986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.781141 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.781166 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 02:12:12.781267 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 02:12:12.781386 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"spinlock_wait_cycles":9344,"thread_start_us":648,"threads_started":1}
I20260929 02:12:12.782878 26867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c9818d1a074d488de2645f9a768b9e" candidate_uuid: "1d92f787d46b45ebb9799645da9013d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9ed8f389a5447388269d6bb541515e" is_pre_election: true
I20260929 02:12:12.783353 26867 raft_consensus.cc:2454] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d92f787d46b45ebb9799645da9013d2 in term 0.
I20260929 02:12:12.783919 26653 leader_election.cc:302] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9ed8f389a5447388269d6bb541515e, 1d92f787d46b45ebb9799645da9013d2; no voters:
I20260929 02:12:12.784351 27076 raft_consensus.cc:2781] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:12:12.784482 27076 raft_consensus.cc:482] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:12:12.784579 27076 raft_consensus.cc:3037] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:12.784933 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.775301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:56367 (local address 127.24.216.131:38255)
0929 02:12:12.775485 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:12.775492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:12.776774 (+ 1282us) server_negotiation.cc:407] Connection header received
0929 02:12:12.777087 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:12.777093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:12.777184 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:12.777251 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:12.778652 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.779874 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.782545 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.783899 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.784048 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.784142 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:12.784563 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 02:12:12.784682 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:12:12.785050 27101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.775323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:38255 (local address 127.24.216.129:56367)
0929 02:12:12.776670 (+ 1347us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:12.776689 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:12.776902 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:12.777522 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:12.777529 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:12.777559 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:12.778432 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.778445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.780060 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.780065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.782364 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.782379 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.784196 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.784204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.784369 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.784390 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 02:12:12.784537 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 02:12:12.784656 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"spinlock_wait_cycles":10240,"thread_start_us":1007,"threads_started":1}
I20260929 02:12:12.785825 27006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c9818d1a074d488de2645f9a768b9e" candidate_uuid: "1d92f787d46b45ebb9799645da9013d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8504c9357884d9483789b7c9361ea68" is_pre_election: true
I20260929 02:12:12.786049 27006 raft_consensus.cc:2454] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d92f787d46b45ebb9799645da9013d2 in term 0.
I20260929 02:12:12.788064 27076 raft_consensus.cc:504] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.789225 27076 leader_election.cc:288] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9ed8f389a5447388269d6bb541515e (127.24.216.130:38709), a8504c9357884d9483789b7c9361ea68 (127.24.216.131:38255)
I20260929 02:12:12.789570 26867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c9818d1a074d488de2645f9a768b9e" candidate_uuid: "1d92f787d46b45ebb9799645da9013d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9ed8f389a5447388269d6bb541515e"
I20260929 02:12:12.789760 26867 raft_consensus.cc:3037] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:12.789848 27006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c9818d1a074d488de2645f9a768b9e" candidate_uuid: "1d92f787d46b45ebb9799645da9013d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8504c9357884d9483789b7c9361ea68"
I20260929 02:12:12.790038 27006 raft_consensus.cc:3037] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:12.792563 26867 raft_consensus.cc:2454] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d92f787d46b45ebb9799645da9013d2 in term 1.
I20260929 02:12:12.792958 26653 leader_election.cc:302] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9ed8f389a5447388269d6bb541515e, 1d92f787d46b45ebb9799645da9013d2; no voters:
I20260929 02:12:12.793202 27076 raft_consensus.cc:2781] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:12.793296 27006 raft_consensus.cc:2454] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d92f787d46b45ebb9799645da9013d2 in term 1.
I20260929 02:12:12.794051 27076 raft_consensus.cc:686] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1d92f787d46b45ebb9799645da9013d2, State: Running, Role: LEADER
I20260929 02:12:12.794425 27076 consensus_queue.cc:237] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.797734 26564 catalog_manager.cc:5495] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d92f787d46b45ebb9799645da9013d2 (127.24.216.129). New cstate: current_term: 1 leader_uuid: "1d92f787d46b45ebb9799645da9013d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } health_report { overall_health: UNKNOWN } } }
Sep 29 02:12:12 dist-test-slave-9k0h krb5kdc[26520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647930, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:12.862586 27109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.848348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:35131 (local address 127.0.0.1:35682)
0929 02:12:12.849030 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:12.849052 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:12.849276 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:12.849670 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:12.849674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:12.850493 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:12.850759 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.850775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.851802 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.851807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.852785 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:12.852796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.853495 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.853500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:12.853664 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.854382 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:12.854398 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:12.857083 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:12.860165 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:12.860179 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:12.860186 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:12.860582 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:12.860924 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:12.860930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:12.860934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:12.861050 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:12.861519 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:12.861527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:12.861690 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 02:12:12.861865 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 02:12:12.862127 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":361,"threads_started":1}
I20260929 02:12:12.862617 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:12.848381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.24.216.190:35131)
0929 02:12:12.849406 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:12.849411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:12.849426 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:12:12.849459 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:12.849462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:12.849551 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:12.849653 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:12.850922 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.851662 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.852918 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:12.853332 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:12.853478 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:12.857342 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:12.857349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:12.857356 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:12.857390 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:12.860046 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:12.860693 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:12.860702 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:12.860705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:12.860823 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:12.861145 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:12.861152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:12.861155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:12.861400 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:12.861544 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:12.861934 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 02:12:12.862260 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":881,"thread_start_us":672,"threads_started":1}
I20260929 02:12:12.867889 26564 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:35682:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:12:12.868904 26564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:12:12.876403 26702 tablet_service.cc:1467] Processing CreateTablet for tablet eea13e2660af40c2900124e68496bd9c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e938bf1af37a4ed9b9907e26c86c32a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:12:12.876920 26702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea13e2660af40c2900124e68496bd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:12.876777 26842 tablet_service.cc:1467] Processing CreateTablet for tablet eea13e2660af40c2900124e68496bd9c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e938bf1af37a4ed9b9907e26c86c32a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:12:12.877358 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea13e2660af40c2900124e68496bd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:12.881872 27071 tablet_bootstrap.cc:492] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2: Bootstrap starting.
I20260929 02:12:12.882963 26980 tablet_service.cc:1467] Processing CreateTablet for tablet eea13e2660af40c2900124e68496bd9c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e938bf1af37a4ed9b9907e26c86c32a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:12:12.883462 26980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea13e2660af40c2900124e68496bd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:12.883620 27071 tablet_bootstrap.cc:654] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:12.888363 27072 tablet_bootstrap.cc:492] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e: Bootstrap starting.
I20260929 02:12:12.890106 27072 tablet_bootstrap.cc:654] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:12.890987 27070 tablet_bootstrap.cc:492] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68: Bootstrap starting.
I20260929 02:12:12.892759 27070 tablet_bootstrap.cc:654] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:12.899237 27071 tablet_bootstrap.cc:492] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2: No bootstrap required, opened a new log
I20260929 02:12:12.899432 27071 ts_tablet_manager.cc:1397] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20260929 02:12:12.900168 27071 raft_consensus.cc:348] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.900430 27071 raft_consensus.cc:374] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:12.900503 27071 raft_consensus.cc:729] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d92f787d46b45ebb9799645da9013d2, State: Initialized, Role: FOLLOWER
I20260929 02:12:12.900792 27071 consensus_queue.cc:260] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.901273 27076 tablet_replica.cc:439] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } }
I20260929 02:12:12.901549 27076 tablet_replica.cc:442] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:12:12.902741 27071 ts_tablet_manager.cc:1428] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 02:12:12.903505 27070 tablet_bootstrap.cc:492] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68: No bootstrap required, opened a new log
I20260929 02:12:12.903666 27070 ts_tablet_manager.cc:1397] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.009s
I20260929 02:12:12.903862 27072 tablet_bootstrap.cc:492] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e: No bootstrap required, opened a new log
I20260929 02:12:12.904112 27072 ts_tablet_manager.cc:1397] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20260929 02:12:12.904455 27070 raft_consensus.cc:348] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.904701 27070 raft_consensus.cc:374] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:12.904793 27070 raft_consensus.cc:729] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8504c9357884d9483789b7c9361ea68, State: Initialized, Role: FOLLOWER
I20260929 02:12:12.905078 27070 consensus_queue.cc:260] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.905329 27072 raft_consensus.cc:348] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.905678 27072 raft_consensus.cc:374] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:12.905490 27077 tablet_replica.cc:439] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } }
I20260929 02:12:12.905819 27077 tablet_replica.cc:442] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:12:12.905821 27072 raft_consensus.cc:729] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9ed8f389a5447388269d6bb541515e, State: Initialized, Role: FOLLOWER
I20260929 02:12:12.906620 27070 ts_tablet_manager.cc:1428] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 02:12:12.906404 27072 consensus_queue.cc:260] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.907209 27072 ts_tablet_manager.cc:1428] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 02:12:12.907007 27089 tablet_replica.cc:439] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } }
I20260929 02:12:12.907402 27089 tablet_replica.cc:442] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:12:12.971297 27076 raft_consensus.cc:482] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:12:12.971526 27076 raft_consensus.cc:504] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.972138 27076 leader_election.cc:288] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a9ed8f389a5447388269d6bb541515e (127.24.216.130:38709), a8504c9357884d9483789b7c9361ea68 (127.24.216.131:38255)
I20260929 02:12:12.972685 26867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea13e2660af40c2900124e68496bd9c" candidate_uuid: "1d92f787d46b45ebb9799645da9013d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9ed8f389a5447388269d6bb541515e" is_pre_election: true
I20260929 02:12:12.972739 27006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea13e2660af40c2900124e68496bd9c" candidate_uuid: "1d92f787d46b45ebb9799645da9013d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8504c9357884d9483789b7c9361ea68" is_pre_election: true
I20260929 02:12:12.972955 26867 raft_consensus.cc:2454] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d92f787d46b45ebb9799645da9013d2 in term 0.
I20260929 02:12:12.973387 26653 leader_election.cc:302] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9ed8f389a5447388269d6bb541515e, 1d92f787d46b45ebb9799645da9013d2; no voters:
I20260929 02:12:12.973449 27006 raft_consensus.cc:2454] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d92f787d46b45ebb9799645da9013d2 in term 0.
I20260929 02:12:12.973676 27076 raft_consensus.cc:2781] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:12:12.973781 27076 raft_consensus.cc:482] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:12:12.973901 27076 raft_consensus.cc:3037] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:12.975804 27076 raft_consensus.cc:504] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.976315 27076 leader_election.cc:288] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a9ed8f389a5447388269d6bb541515e (127.24.216.130:38709), a8504c9357884d9483789b7c9361ea68 (127.24.216.131:38255)
I20260929 02:12:12.976667 26867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea13e2660af40c2900124e68496bd9c" candidate_uuid: "1d92f787d46b45ebb9799645da9013d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a9ed8f389a5447388269d6bb541515e"
I20260929 02:12:12.976800 27006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea13e2660af40c2900124e68496bd9c" candidate_uuid: "1d92f787d46b45ebb9799645da9013d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8504c9357884d9483789b7c9361ea68"
I20260929 02:12:12.976903 26867 raft_consensus.cc:3037] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:12.977008 27006 raft_consensus.cc:3037] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:12.979875 27006 raft_consensus.cc:2454] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d92f787d46b45ebb9799645da9013d2 in term 1.
I20260929 02:12:12.980140 26867 raft_consensus.cc:2454] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d92f787d46b45ebb9799645da9013d2 in term 1.
I20260929 02:12:12.980326 26656 leader_election.cc:302] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d92f787d46b45ebb9799645da9013d2, a8504c9357884d9483789b7c9361ea68; no voters:
I20260929 02:12:12.980620 27076 raft_consensus.cc:2781] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:12.980815 27076 raft_consensus.cc:686] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1d92f787d46b45ebb9799645da9013d2, State: Running, Role: LEADER
I20260929 02:12:12.981051 27076 consensus_queue.cc:237] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } }
I20260929 02:12:12.989807 27102 tablet_replica.cc:439] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2: TxnStatusTablet state changed. Reason: New leader 1d92f787d46b45ebb9799645da9013d2. Latest consensus state: current_term: 1 leader_uuid: "1d92f787d46b45ebb9799645da9013d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } }
I20260929 02:12:12.990223 27102 tablet_replica.cc:442] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:12:12.991017 26564 catalog_manager.cc:5495] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d92f787d46b45ebb9799645da9013d2 (127.24.216.129). New cstate: current_term: 1 leader_uuid: "1d92f787d46b45ebb9799645da9013d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } health_report { overall_health: UNKNOWN } } }
I20260929 02:12:12.991394 27115 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:12:12.991668 27115 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:12:13.026214 27109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:13.013458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:41669 (local address 127.0.0.1:40798)
0929 02:12:13.013686 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:13.013705 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:13.013822 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:13.019069 (+ 5247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:13.019074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:13.019089 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:12:13.019297 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:13.019310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.021485 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.021491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:13.023126 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:13.023140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.024094 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.024100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:13.024315 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:13.024384 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:12:13.025626 (+ 1242us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:12:13.025648 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:12:13.025741 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 02:12:13.025895 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 02:12:13.026243 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:13.013443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40798 (local address 127.24.216.129:41669)
0929 02:12:13.018624 (+ 5181us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:13.018632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:13.018656 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:12:13.018752 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:13.018757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:13.018861 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:13.019023 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:12:13.019477 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.021304 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.023288 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.023926 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.024046 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:13.024608 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:12:13.025446 (+ 838us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:12:13.025667 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:13.025762 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 02:12:13.025842 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5018,"thread_start_us":4863,"threads_started":1}
I20260929 02:12:13.031150 27006 raft_consensus.cc:1261] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68 [term 1 FOLLOWER]: Refusing update from remote peer 1d92f787d46b45ebb9799645da9013d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:13.031324 26867 raft_consensus.cc:1261] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e [term 1 FOLLOWER]: Refusing update from remote peer 1d92f787d46b45ebb9799645da9013d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:13.032019 27076 consensus_queue.cc:1035] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:13.032712 27102 consensus_queue.cc:1035] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:13.046604 27077 tablet_replica.cc:439] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68: TxnStatusTablet state changed. Reason: New leader 1d92f787d46b45ebb9799645da9013d2. Latest consensus state: current_term: 1 leader_uuid: "1d92f787d46b45ebb9799645da9013d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } }
I20260929 02:12:13.046887 27077 tablet_replica.cc:442] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:12:13.048382 27089 tablet_replica.cc:439] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e: TxnStatusTablet state changed. Reason: New leader 1d92f787d46b45ebb9799645da9013d2. Latest consensus state: current_term: 1 leader_uuid: "1d92f787d46b45ebb9799645da9013d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } }
I20260929 02:12:13.048664 27089 tablet_replica.cc:442] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:12:13.050258 27076 tablet_replica.cc:439] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d92f787d46b45ebb9799645da9013d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } }
I20260929 02:12:13.050539 27076 tablet_replica.cc:442] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:12:13.050792 27076 tablet_replica.cc:439] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d92f787d46b45ebb9799645da9013d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } }
I20260929 02:12:13.051046 27076 tablet_replica.cc:442] T eea13e2660af40c2900124e68496bd9c P 1d92f787d46b45ebb9799645da9013d2: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:12:13.053185 27077 tablet_replica.cc:439] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d92f787d46b45ebb9799645da9013d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } }
I20260929 02:12:13.053438 27077 tablet_replica.cc:442] T eea13e2660af40c2900124e68496bd9c P a8504c9357884d9483789b7c9361ea68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:12:13.067418 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:13.059759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.24.216.129:41669)
0929 02:12:13.059907 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:13.059913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:13.059930 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:12:13.059988 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:13.060000 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:13.060072 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:13.060589 (+ 517us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:12:13.060720 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.061947 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.064014 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.065135 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.065757 (+ 622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:13.065863 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:12:13.065994 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:12:13.066993 (+ 999us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:13.067041 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 02:12:13.067127 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:12:13.088187 27089 tablet_replica.cc:439] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d92f787d46b45ebb9799645da9013d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d92f787d46b45ebb9799645da9013d2" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41669 } } peers { permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 } } peers { permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 } } }
I20260929 02:12:13.088482 27089 tablet_replica.cc:442] T eea13e2660af40c2900124e68496bd9c P 1a9ed8f389a5447388269d6bb541515e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:12:13.119452 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:13.100373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:36099 (local address 127.24.216.129:41669)
0929 02:12:13.100578 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:13.100585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:13.111837 (+ 11252us) server_negotiation.cc:407] Connection header received
0929 02:12:13.112335 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:13.112340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:13.112436 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:13.112517 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:12:13.113245 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.114475 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.116763 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.117424 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.117521 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:13.118174 (+ 653us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:12:13.118302 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:12:13.118363 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:13.118594 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 02:12:13.119102 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:12:13.120369 27123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:13.099950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:41669 (local address 127.24.216.129:36099)
0929 02:12:13.111736 (+ 11786us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:13.111758 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:13.112156 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:13.112699 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:13.112703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:13.112714 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:12:13.113076 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:13.113087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.114638 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.114644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:13.116620 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:13.116635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.117695 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.117701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:13.117883 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:13.117952 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:12:13.118484 (+ 532us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:12:13.118493 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:12:13.119786 (+ 1293us) client_negotiation.cc:240] Negotiation successful
0929 02:12:13.119950 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11145,"thread_start_us":11895,"threads_started":1}
I20260929 02:12:13.133744 26867 raft_consensus.cc:1261] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e [term 1 FOLLOWER]: Refusing update from remote peer 1d92f787d46b45ebb9799645da9013d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:13.134428 27102 consensus_queue.cc:1035] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a9ed8f389a5447388269d6bb541515e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:13.143760 27006 raft_consensus.cc:1261] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68 [term 1 FOLLOWER]: Refusing update from remote peer 1d92f787d46b45ebb9799645da9013d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:13.145087 27076 consensus_queue.cc:1035] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8504c9357884d9483789b7c9361ea68" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:13.225616 26564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:13.225920 26564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:13.228888 26564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42b28fed4c3f422da79979f3fda0941e: Sending DeleteTablet for 3 replicas of tablet 98c9818d1a074d488de2645f9a768b9e
I20260929 02:12:13.229642 26703 tablet_service.cc:1514] Processing DeleteTablet for tablet 98c9818d1a074d488de2645f9a768b9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:13 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:40790
I20260929 02:12:13.229950 26842 tablet_service.cc:1514] Processing DeleteTablet for tablet 98c9818d1a074d488de2645f9a768b9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:13 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:50618
I20260929 02:12:13.230481 26980 tablet_service.cc:1514] Processing DeleteTablet for tablet 98c9818d1a074d488de2645f9a768b9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:13 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:37092
I20260929 02:12:13.234534 27130 tablet_replica.cc:330] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e: stopping tablet replica
I20260929 02:12:13.234805 27130 raft_consensus.cc:2229] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:13.235090 27130 raft_consensus.cc:2258] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:13.236157 27130 ts_tablet_manager.cc:1905] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:13.243340 27130 ts_tablet_manager.cc:1918] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:12:13.243556 27130 log.cc:1196] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/98c9818d1a074d488de2645f9a768b9e
I20260929 02:12:13.244143 27130 ts_tablet_manager.cc:1939] T 98c9818d1a074d488de2645f9a768b9e P 1a9ed8f389a5447388269d6bb541515e: Deleting consensus metadata
I20260929 02:12:13.245251 26549 catalog_manager.cc:4841] TS 1a9ed8f389a5447388269d6bb541515e (127.24.216.130:38709): tablet 98c9818d1a074d488de2645f9a768b9e (table test-table [id=2e19d64e8e8847549d5fd28148aa5eb8]) successfully deleted
I20260929 02:12:13.246773 27131 tablet_replica.cc:330] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68: stopping tablet replica
I20260929 02:12:13.247035 27131 raft_consensus.cc:2229] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:13.247366 27131 raft_consensus.cc:2258] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:13.248046 27129 tablet_replica.cc:330] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2: stopping tablet replica
I20260929 02:12:13.248301 27129 raft_consensus.cc:2229] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:12:13.248926 27129 raft_consensus.cc:2258] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:13.249054 27131 ts_tablet_manager.cc:1905] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:13.249984 27129 ts_tablet_manager.cc:1905] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:13.257156 27131 ts_tablet_manager.cc:1918] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:12:13.257302 27131 log.cc:1196] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/98c9818d1a074d488de2645f9a768b9e
I20260929 02:12:13.257916 27129 ts_tablet_manager.cc:1918] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:12:13.258091 27129 log.cc:1196] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/98c9818d1a074d488de2645f9a768b9e
I20260929 02:12:13.258122 27131 ts_tablet_manager.cc:1939] T 98c9818d1a074d488de2645f9a768b9e P a8504c9357884d9483789b7c9361ea68: Deleting consensus metadata
I20260929 02:12:13.258832 27129 ts_tablet_manager.cc:1939] T 98c9818d1a074d488de2645f9a768b9e P 1d92f787d46b45ebb9799645da9013d2: Deleting consensus metadata
I20260929 02:12:13.259464 26552 catalog_manager.cc:4841] TS a8504c9357884d9483789b7c9361ea68 (127.24.216.131:38255): tablet 98c9818d1a074d488de2645f9a768b9e (table test-table [id=2e19d64e8e8847549d5fd28148aa5eb8]) successfully deleted
I20260929 02:12:13.259974 26552 catalog_manager.cc:4841] TS 1d92f787d46b45ebb9799645da9013d2 (127.24.216.129:41669): tablet 98c9818d1a074d488de2645f9a768b9e (table test-table [id=2e19d64e8e8847549d5fd28148aa5eb8]) successfully deleted
Sep 29 02:12:13 dist-test-slave-9k0h krb5kdc[26520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:12:13.269260 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:13.247416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40826 (local address 127.24.216.129:41669)
0929 02:12:13.247647 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:13.247653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:13.251184 (+ 3531us) server_negotiation.cc:407] Connection header received
0929 02:12:13.251662 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:13.251667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:13.251749 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:13.251835 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:13.253918 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.255123 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.257194 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.258544 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.258649 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:13.263224 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:13.263235 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:13.263250 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:13.263290 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:13.266344 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:13.267108 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:13.267115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:13.267118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:13.267301 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:13.267739 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:13.267743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:13.267745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:13.268197 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:13.268500 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:13.268772 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 02:12:13.268941 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 02:12:13.270012 26768 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40826
I20260929 02:12:13.287410 27110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:13.272878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.24.216.190:35131)
0929 02:12:13.273054 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:13.273060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:13.273700 (+ 640us) server_negotiation.cc:407] Connection header received
0929 02:12:13.273871 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:13.273875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:13.273935 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:13.274013 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:13.275685 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.276370 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.278211 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.278615 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.278704 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:13.281479 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:13.281483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:13.281487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:13.281515 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:13.284250 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:13.285537 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:13.285541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:13.285543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:13.285656 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:13.286038 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:13.286041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:13.286043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:13.286403 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:13.286588 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:13.286913 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 02:12:13.287127 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 02:12:13.288026 26564 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35690
I20260929 02:12:13.290138 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 26637
I20260929 02:12:13.314390 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:13.291429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.24.216.129:41669)
0929 02:12:13.291658 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:13.291666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:13.303004 (+ 11338us) server_negotiation.cc:407] Connection header received
0929 02:12:13.303188 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:13.303191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:13.303248 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:13.303352 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:13.304537 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.305203 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.306417 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.306871 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.307004 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:13.309311 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:13.309315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:13.309318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:13.309347 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:13.311955 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:13.312563 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:13.312570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:13.312578 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:13.312709 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:13.313054 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:13.313060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:13.313068 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:13.313316 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:13.313593 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:13.313861 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 02:12:13.314059 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 02:12:13.314880 26768 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40832
W20260929 02:12:13.315500 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 26637: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:12:13.315667 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 26637
I20260929 02:12:13.336061 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 26777
Sep 29 02:12:13 dist-test-slave-9k0h krb5kdc[26520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:12:13.349726 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:13.336841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50630 (local address 127.24.216.130:38709)
0929 02:12:13.337228 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:13.337232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:13.337248 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:12:13.337308 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:13.337312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:13.337384 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:13.337475 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:13.338609 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.339320 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.340443 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.340861 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.340960 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:13.344544 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:13.344554 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:13.344568 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:13.344603 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:13.347396 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:13.347930 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:13.347936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:13.347938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:13.348059 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:13.348371 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:13.348374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:13.348376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:13.348693 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:13.348976 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:13.349162 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 02:12:13.349343 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":204,"threads_started":1}
W20260929 02:12:13.350473 26907 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50630
W20260929 02:12:13.351074 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 26777: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:12:13.351241 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 26777
I20260929 02:12:13.372359 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 26916
Sep 29 02:12:13 dist-test-slave-9k0h krb5kdc[26520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:12:13.386729 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:13.373084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.24.216.131:38255)
0929 02:12:13.373583 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:13.373587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:13.373603 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:12:13.373674 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:13.373678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:13.373746 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:13.373837 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:13.375013 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.375707 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.376905 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:13.377344 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:13.377431 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:13.381902 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:13.381915 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:13.381931 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:13.381965 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:13.384541 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:13.385080 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:13.385087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:13.385089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:13.385202 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:13.385475 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:13.385478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:13.385480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:13.385763 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:13.385969 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:13.386335 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 02:12:13.386495 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":294,"threads_started":1}
W20260929 02:12:13.387405 27046 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37104
W20260929 02:12:13.388027 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 26916: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:12:13.388168 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 26916
I20260929 02:12:13.405444 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 26533
I20260929 02:12:13.406525 26605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:13.517966 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 26533
2026-09-29T02:12:13Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4688 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:12:13 dist-test-slave-9k0h krb5kdc[27142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:12:13 dist-test-slave-9k0h krb5kdc[27142](info): set up 2 sockets
Sep 29 02:12:13 dist-test-slave-9k0h krb5kdc[27142](info): commencing operation
krb5kdc: starting...
W20260929 02:12:14.582386 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:12:14 dist-test-slave-9k0h krb5kdc[27142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:12:14.761544 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:36893
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:36893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:15.070446 27155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:15.070761 27155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:15.070933 27155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:15.077929 27155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:15.078058 27155 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:12:15.078152 27155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:15.078291 27155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:15.078377 27155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:15.078461 27155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:12:15.087932 27155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:36893
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:36893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:15.089157 27155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:15.090461 27155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:15.101958 27161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:15.102072 27160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:15.102034 27163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:15.103178 27155 server_base.cc:1034] running on GCE node
I20260929 02:12:15.103825 27155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:15.105019 27155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:15.106267 27155 hybrid_clock.cc:648] HybridClock initialized: now 1790647935106201 us; error 50 us; skew 500 ppm
I20260929 02:12:15.106586 27155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:15 dist-test-slave-9k0h krb5kdc[27142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647935, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:15.109529 27155 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:12:15.110993 27155 webserver.cc:460] Webserver started at http://127.24.216.190:39281/ using document root <none> and password file <none>
I20260929 02:12:15.111658 27155 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:15.111786 27155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:15.112097 27155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:15.114940 27155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "37b5d010f8514d60adcbd12e7f4321f9"
format_stamp: "Formatted at 2026-09-29 02:12:15 on dist-test-slave-9k0h"
server_key: "94ffa67f6f2f8c0d96e7f5c7bf7c1bf7"
server_key_iv: "f55126f27097cf326eca55f314018b60"
server_key_version: "encryptionkey@0"
I20260929 02:12:15.115684 27155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "37b5d010f8514d60adcbd12e7f4321f9"
format_stamp: "Formatted at 2026-09-29 02:12:15 on dist-test-slave-9k0h"
server_key: "94ffa67f6f2f8c0d96e7f5c7bf7c1bf7"
server_key_iv: "f55126f27097cf326eca55f314018b60"
server_key_version: "encryptionkey@0"
I20260929 02:12:15.120610 27155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:12:15.123808 27170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:15.125056 27155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:12:15.125228 27155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "37b5d010f8514d60adcbd12e7f4321f9"
format_stamp: "Formatted at 2026-09-29 02:12:15 on dist-test-slave-9k0h"
server_key: "94ffa67f6f2f8c0d96e7f5c7bf7c1bf7"
server_key_iv: "f55126f27097cf326eca55f314018b60"
server_key_version: "encryptionkey@0"
I20260929 02:12:15.125434 27155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:15.145444 27155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:15.149617 27155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:15.149896 27155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:15.171046 27155 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:36893
I20260929 02:12:15.171064 27231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:36893 every 8 connection(s)
I20260929 02:12:15.172421 27155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:15.176012 27232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:15.178613 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 27155
I20260929 02:12:15.178823 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:15.179605 25442 external_mini_cluster.cc:1428] Setting key bed58c554505a627bccddfed955631dd
I20260929 02:12:15.188344 27232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9: Bootstrap starting.
I20260929 02:12:15.192134 27232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:15.193444 27232 log.cc:824] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:12:15 dist-test-slave-9k0h krb5kdc[27142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:15.196251 27232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9: No bootstrap required, opened a new log
I20260929 02:12:15.201749 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:15.181648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.24.216.190:36893)
0929 02:12:15.182410 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:15.182421 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:15.182459 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:12:15.184003 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:15.184025 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:15.184391 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:15.184699 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:15.186403 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:15.187637 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:15.189288 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:15.189953 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:15.190076 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:15.194958 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:15.194965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:15.194979 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:15.195009 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:15.197916 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:15.198629 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:15.198635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:15.198640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:15.198801 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:15.199169 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:15.199173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:15.199179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:15.199665 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:15.199968 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:15.200403 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 02:12:15.200801 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":360,"threads_started":1}
I20260929 02:12:15.203974 27232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b5d010f8514d60adcbd12e7f4321f9" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36893 } }
I20260929 02:12:15.204335 27232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:15.204445 27232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b5d010f8514d60adcbd12e7f4321f9, State: Initialized, Role: FOLLOWER
I20260929 02:12:15.204955 27232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b5d010f8514d60adcbd12e7f4321f9" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36893 } }
I20260929 02:12:15.205150 27232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:15.205243 27232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:15.205365 27232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:15.207428 27232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b5d010f8514d60adcbd12e7f4321f9" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36893 } }
I20260929 02:12:15.207906 27232 leader_election.cc:302] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37b5d010f8514d60adcbd12e7f4321f9; no voters:
I20260929 02:12:15.208575 27232 leader_election.cc:288] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:15.208676 27237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:15.209455 27237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [term 1 LEADER]: Becoming Leader. State: Replica: 37b5d010f8514d60adcbd12e7f4321f9, State: Running, Role: LEADER
I20260929 02:12:15.209993 27237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b5d010f8514d60adcbd12e7f4321f9" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36893 } }
I20260929 02:12:15.210829 27232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:15.212950 27239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37b5d010f8514d60adcbd12e7f4321f9. Latest consensus state: current_term: 1 leader_uuid: "37b5d010f8514d60adcbd12e7f4321f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b5d010f8514d60adcbd12e7f4321f9" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36893 } } }
I20260929 02:12:15.212875 27238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37b5d010f8514d60adcbd12e7f4321f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b5d010f8514d60adcbd12e7f4321f9" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36893 } } }
I20260929 02:12:15.213348 27238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:15.227564 27239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:15.228488 27244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:15.235054 27244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:15.244896 27244 catalog_manager.cc:1347] Generated new cluster ID: 93e797eb2b5749b3a7bf0d8d8a43b147
I20260929 02:12:15.245028 27244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:15.263171 27244 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:15.264546 27244 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:15.273329 27244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9: Generated new TSK 0
I20260929 02:12:15.274245 27244 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:12:15.321861 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36893
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:15.638352 27258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:15.638612 27258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:15.638787 27258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:15.646029 27258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:15.646176 27258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:15.646332 27258 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:12:15.646425 27258 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:12:15.646530 27258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:12:15.655965 27258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.216.190:36893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:15.657205 27258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:15.658565 27258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:15.671200 27264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:15.671247 27263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:15.671272 27266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:15.672055 27258 server_base.cc:1034] running on GCE node
I20260929 02:12:15.672587 27258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:15.673389 27258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:15.674572 27258 hybrid_clock.cc:648] HybridClock initialized: now 1790647935674535 us; error 37 us; skew 500 ppm
I20260929 02:12:15.674834 27258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:15 dist-test-slave-9k0h krb5kdc[27142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647935, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:15.677768 27258 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:12:15.679150 27258 webserver.cc:460] Webserver started at http://127.24.216.129:44273/ using document root <none> and password file <none>
I20260929 02:12:15.679813 27258 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:15.679943 27258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:15.680254 27258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:15.683025 27258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "b2a7f958f73f466d9abacb5a240437df"
format_stamp: "Formatted at 2026-09-29 02:12:15 on dist-test-slave-9k0h"
server_key: "9d994475180d350e11ba9fd6b728983a"
server_key_iv: "95317fd95d13873ad53e000452eccac3"
server_key_version: "encryptionkey@0"
I20260929 02:12:15.683734 27258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "b2a7f958f73f466d9abacb5a240437df"
format_stamp: "Formatted at 2026-09-29 02:12:15 on dist-test-slave-9k0h"
server_key: "9d994475180d350e11ba9fd6b728983a"
server_key_iv: "95317fd95d13873ad53e000452eccac3"
server_key_version: "encryptionkey@0"
I20260929 02:12:15.688551 27258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:12:15.691677 27273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:15.692898 27258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 02:12:15.693069 27258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "b2a7f958f73f466d9abacb5a240437df"
format_stamp: "Formatted at 2026-09-29 02:12:15 on dist-test-slave-9k0h"
server_key: "9d994475180d350e11ba9fd6b728983a"
server_key_iv: "95317fd95d13873ad53e000452eccac3"
server_key_version: "encryptionkey@0"
I20260929 02:12:15.693246 27258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:15.709324 27258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:15.713209 27258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:15.713485 27258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:15.716295 27258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:15.716431 27258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:15.716555 27258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:15.716653 27258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:12:15 dist-test-slave-9k0h krb5kdc[27142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647935, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:15.752442 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:15.717203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36893 (local address 127.24.216.129:32963)
0929 02:12:15.718217 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:15.718278 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:15.721111 (+ 2833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:15.732272 (+ 11161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:15.732288 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:15.733354 (+ 1066us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:15.734513 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:15.734546 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:15.736222 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:15.736229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:15.738493 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:15.738513 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:15.739793 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:15.739798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:15.739991 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:15.740853 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:15.740890 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:15.743912 (+ 3022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:15.747748 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:15.747753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:15.747765 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:15.748363 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:15.748733 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:15.748737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:15.748742 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:15.748930 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:15.749451 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:15.749458 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:15.749853 (+ 395us) client_negotiation.cc:769] Sending connection context
0929 02:12:15.750283 (+ 430us) client_negotiation.cc:240] Negotiation successful
0929 02:12:15.750806 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":439,"threads_started":1}
I20260929 02:12:15.752996 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:15.716484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:32963 (local address 127.24.216.190:36893)
0929 02:12:15.731590 (+ 15106us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:15.731595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:15.731613 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:12:15.731693 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:15.731700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:15.731793 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:15.731929 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:15.734775 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:15.736038 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:15.738660 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:15.739540 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:15.739702 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:15.744173 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:15.744181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:15.744186 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:15.744231 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:15.747594 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:15.748495 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:15.748500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:15.748504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:15.748628 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:15.749049 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:15.749054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:15.749057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:15.749311 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:15.749458 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:15.752455 (+ 2997us) server_negotiation.cc:299] Negotiation successful
0929 02:12:15.752670 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15002,"thread_start_us":15528,"threads_started":1}
I20260929 02:12:15.764382 27258 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:41567
I20260929 02:12:15.764454 27393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:41567 every 8 connection(s)
I20260929 02:12:15.765583 27258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:12:15.767668 27394 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36893
I20260929 02:12:15.767916 27394 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:15.768528 27394 heartbeater.cc:511] Master 127.24.216.190:36893 requested a full tablet report, sending...
I20260929 02:12:15.770093 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 27258
I20260929 02:12:15.770346 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:12:15.770710 27187 ts_manager.cc:194] Registered new tserver with Master: b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567)
I20260929 02:12:15.771154 25442 external_mini_cluster.cc:1428] Setting key b7b36e5f32271f243b90b5fc9d02b210
I20260929 02:12:15.771425 27387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:15.758884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36893 (local address 127.24.216.129:60987)
0929 02:12:15.759547 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:15.759571 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:15.759764 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:15.760116 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:15.760120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:15.760517 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:15.760797 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:15.760805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:15.761747 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:15.761755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:15.762818 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:15.762829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:15.763789 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:15.763795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:15.763930 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:15.764626 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:15.764648 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:15.765305 (+ 657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:15.768608 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:15.768615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:15.768618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:15.768998 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:15.769435 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:15.769439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:15.769442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:15.769594 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:15.770281 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:15.770287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:15.770488 (+ 201us) client_negotiation.cc:769] Sending connection context
0929 02:12:15.770661 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 02:12:15.770981 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":334,"threads_started":1}
I20260929 02:12:15.771445 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:15.758881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:60987 (local address 127.24.216.190:36893)
0929 02:12:15.759091 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:15.759100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:15.759657 (+ 557us) server_negotiation.cc:407] Connection header received
0929 02:12:15.759924 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:15.759930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:15.759997 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:15.760090 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:15.760931 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:15.761622 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:15.762956 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:15.763424 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:15.763556 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:15.765472 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:15.765477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:15.765480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:15.765504 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:15.768488 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:15.769130 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:15.769138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:15.769142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:15.769317 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:15.769713 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:15.769720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:15.769725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:15.770075 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:15.770308 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:15.770778 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 02:12:15.771113 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:12:15.772651 27187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:32963
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:12:15.809558 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36893
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:16.117401 27398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:16.117702 27398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:16.117901 27398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:16.124990 27398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:16.125133 27398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:16.125293 27398 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:12:16.125380 27398 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:12:16.125509 27398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:12:16.135471 27398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.216.190:36893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:16.136791 27398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:16.138146 27398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:16.151111 27406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:16.151222 27403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:16.151356 27404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:16.152587 27398 server_base.cc:1034] running on GCE node
I20260929 02:12:16.153156 27398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:16.154105 27398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:16.155336 27398 hybrid_clock.cc:648] HybridClock initialized: now 1790647936155289 us; error 42 us; skew 500 ppm
I20260929 02:12:16.155596 27398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:16 dist-test-slave-9k0h krb5kdc[27142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647936, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:16.158690 27398 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:12:16.160066 27398 webserver.cc:460] Webserver started at http://127.24.216.130:35011/ using document root <none> and password file <none>
I20260929 02:12:16.160732 27398 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:16.160876 27398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:16.161185 27398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:16.164100 27398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "20f5c0bd7a67469bbd322302a0f410fc"
format_stamp: "Formatted at 2026-09-29 02:12:16 on dist-test-slave-9k0h"
server_key: "7d26717568f2067e910d89ea65e3cf75"
server_key_iv: "d8d771b1034c2018dedeb7d77af18c6f"
server_key_version: "encryptionkey@0"
I20260929 02:12:16.165027 27398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "20f5c0bd7a67469bbd322302a0f410fc"
format_stamp: "Formatted at 2026-09-29 02:12:16 on dist-test-slave-9k0h"
server_key: "7d26717568f2067e910d89ea65e3cf75"
server_key_iv: "d8d771b1034c2018dedeb7d77af18c6f"
server_key_version: "encryptionkey@0"
I20260929 02:12:16.170035 27398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:12:16.173552 27413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:16.174975 27398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:16.175124 27398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "20f5c0bd7a67469bbd322302a0f410fc"
format_stamp: "Formatted at 2026-09-29 02:12:16 on dist-test-slave-9k0h"
server_key: "7d26717568f2067e910d89ea65e3cf75"
server_key_iv: "d8d771b1034c2018dedeb7d77af18c6f"
server_key_version: "encryptionkey@0"
I20260929 02:12:16.175307 27398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:16.197260 27398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:16.201485 27398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:16.201766 27398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:16.204202 27398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:16.204316 27398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:16.204413 27398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:16.204492 27398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:12:16 dist-test-slave-9k0h krb5kdc[27142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647936, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:16.229184 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.205720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36893 (local address 127.24.216.130:48207)
0929 02:12:16.206542 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:16.206577 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:16.208358 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:16.209101 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:16.209117 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:16.209847 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:16.210748 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.210787 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.212473 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.212478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.214561 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.214579 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.215747 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.215759 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.215955 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.216787 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:16.216808 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:16.220550 (+ 3742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:16.224592 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:16.224603 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:16.224619 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:16.225179 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:16.225613 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:16.225619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:16.225626 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:16.225893 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:16.226576 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:16.226583 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:16.227117 (+ 534us) client_negotiation.cc:769] Sending connection context
0929 02:12:16.227453 (+ 336us) client_negotiation.cc:240] Negotiation successful
0929 02:12:16.227905 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":379,"threads_started":1}
I20260929 02:12:16.230086 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.204882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:48207 (local address 127.24.216.190:36893)
0929 02:12:16.205053 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:16.205060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:16.206684 (+ 1624us) server_negotiation.cc:407] Connection header received
0929 02:12:16.208643 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:16.208649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:16.208733 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:16.208840 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:16.210988 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.212302 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.214759 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.215451 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.215609 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.220822 (+ 5213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:16.220849 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:16.220854 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:16.220896 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:16.224443 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:16.225342 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:16.225345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:16.225348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:16.225451 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:16.226015 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:16.226019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:16.226023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:16.226433 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:16.226602 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:16.229731 (+ 3129us) server_negotiation.cc:299] Negotiation successful
0929 02:12:16.229878 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 02:12:16.248997 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.235386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:42161 (local address 127.24.216.190:36893)
0929 02:12:16.235582 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:16.235591 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:16.236386 (+ 795us) server_negotiation.cc:407] Connection header received
0929 02:12:16.236615 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:16.236619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:16.236688 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:16.236762 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:16.238127 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.238977 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.241026 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.241769 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.241916 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.243703 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:16.243709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:16.243713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:16.243750 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:16.246262 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:16.247167 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:16.247175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:16.247181 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:16.247320 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:16.247665 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:16.247671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:16.247675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:16.247955 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:16.248100 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:16.248477 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 02:12:16.248678 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:12:16.249114 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.235371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36893 (local address 127.24.216.130:42161)
0929 02:12:16.236262 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:16.236287 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:16.236452 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:16.236826 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:16.236833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:16.237551 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:16.237988 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.238003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.239175 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.239182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.240824 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.240855 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.241969 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.241975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.242086 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.242761 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:16.242784 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:16.243506 (+ 722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:16.246397 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:16.246402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:16.246406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:16.247046 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:16.247426 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:16.247432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:16.247436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:16.247565 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:16.248067 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:16.248071 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:16.248236 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 02:12:16.248383 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 02:12:16.248678 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":513,"threads_started":1}
I20260929 02:12:16.258378 27398 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:38199
I20260929 02:12:16.258412 27532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:38199 every 8 connection(s)
I20260929 02:12:16.259599 27398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:12:16.261475 27533 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36893
I20260929 02:12:16.261700 27533 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:16.262346 27533 heartbeater.cc:511] Master 127.24.216.190:36893 requested a full tablet report, sending...
I20260929 02:12:16.263521 27187 ts_manager.cc:194] Registered new tserver with Master: 20f5c0bd7a67469bbd322302a0f410fc (127.24.216.130:38199)
I20260929 02:12:16.264269 27187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:48207
I20260929 02:12:16.267275 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 27398
I20260929 02:12:16.267445 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:12:16.267882 25442 external_mini_cluster.cc:1428] Setting key 570c5b5f42d82c54bb27a3c04fc9e55f
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:12:16.305584 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36893
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:16.618584 27537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:16.618870 27537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:16.619042 27537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:16.625910 27537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:16.626051 27537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:16.626221 27537 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:12:16.626317 27537 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:12:16.626430 27537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:12:16.635649 27537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.216.190:36893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:16.636901 27537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:16.638182 27537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:16.651044 27543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:16.651122 27545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:16.651094 27542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:16.652231 27537 server_base.cc:1034] running on GCE node
I20260929 02:12:16.652766 27537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:16.653599 27537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:16.654827 27537 hybrid_clock.cc:648] HybridClock initialized: now 1790647936654779 us; error 50 us; skew 500 ppm
I20260929 02:12:16.655128 27537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:16 dist-test-slave-9k0h krb5kdc[27142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647936, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:16.663394 27537 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:12:16.664806 27537 webserver.cc:460] Webserver started at http://127.24.216.131:42421/ using document root <none> and password file <none>
I20260929 02:12:16.665556 27537 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:16.665694 27537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:16.666015 27537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:16.668804 27537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "d658ff37f51648c2ae3111d2e81aae83"
format_stamp: "Formatted at 2026-09-29 02:12:16 on dist-test-slave-9k0h"
server_key: "38488e36cf04571c20569e714241830d"
server_key_iv: "3e5c339d755e0a24560ad74f484ce177"
server_key_version: "encryptionkey@0"
I20260929 02:12:16.669513 27537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "d658ff37f51648c2ae3111d2e81aae83"
format_stamp: "Formatted at 2026-09-29 02:12:16 on dist-test-slave-9k0h"
server_key: "38488e36cf04571c20569e714241830d"
server_key_iv: "3e5c339d755e0a24560ad74f484ce177"
server_key_version: "encryptionkey@0"
I20260929 02:12:16.674759 27537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:12:16.677994 27552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:16.679405 27537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:12:16.679556 27537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "d658ff37f51648c2ae3111d2e81aae83"
format_stamp: "Formatted at 2026-09-29 02:12:16 on dist-test-slave-9k0h"
server_key: "38488e36cf04571c20569e714241830d"
server_key_iv: "3e5c339d755e0a24560ad74f484ce177"
server_key_version: "encryptionkey@0"
I20260929 02:12:16.679744 27537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:16.694494 27537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:16.698297 27537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:16.698585 27537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:16.701177 27537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:16.701323 27537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:16.701447 27537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:16.701555 27537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:12:16 dist-test-slave-9k0h krb5kdc[27142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647936, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:16.726264 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.701821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:36589 (local address 127.24.216.190:36893)
0929 02:12:16.701998 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:16.702003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:16.703388 (+ 1385us) server_negotiation.cc:407] Connection header received
0929 02:12:16.706399 (+ 3011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:16.706405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:16.706510 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:16.706612 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:16.708974 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.709669 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.713084 (+ 3415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.713650 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.713786 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.718939 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:16.718946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:16.718953 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:16.718996 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:16.722362 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:16.723465 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:16.723472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:16.723477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:16.723633 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:16.724194 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:16.724202 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:16.724208 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:16.724639 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:16.724830 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:16.725461 (+ 631us) server_negotiation.cc:299] Negotiation successful
0929 02:12:16.725757 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:12:16.726833 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.702380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36893 (local address 127.24.216.131:36589)
0929 02:12:16.703203 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:16.703258 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:16.706012 (+ 2754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:16.706917 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:16.706931 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:16.707776 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:16.708715 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.708742 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.709808 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.709816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.712916 (+ 3100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.712935 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.713934 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.713947 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.714329 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.715234 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:16.715280 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:16.718698 (+ 3418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:16.722509 (+ 3811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:16.722522 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:16.722543 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:16.723308 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:16.723742 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:16.723747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:16.723750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:16.724041 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:16.724785 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:16.724791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:16.725080 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 02:12:16.725285 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 02:12:16.725618 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":333,"threads_started":1}
I20260929 02:12:16.746380 27617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.732664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36893 (local address 127.24.216.131:44553)
0929 02:12:16.733289 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:16.733308 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:16.733403 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:16.733707 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:16.733710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:16.734062 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:16.734461 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.734470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.735431 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.735439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.736428 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.736438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.737125 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.737131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.737295 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.737900 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:16.737916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:16.738798 (+ 882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:16.743218 (+ 4420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:16.743225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:16.743234 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:16.743721 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:16.744194 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:16.744204 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:16.744207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:16.744341 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:16.744966 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:16.744972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:16.745220 (+ 248us) client_negotiation.cc:769] Sending connection context
0929 02:12:16.745400 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 02:12:16.745714 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":346,"threads_started":1}
I20260929 02:12:16.746791 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.732652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:44553 (local address 127.24.216.190:36893)
0929 02:12:16.732871 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:16.732880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:16.733380 (+ 500us) server_negotiation.cc:407] Connection header received
0929 02:12:16.733487 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:16.733495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:16.733580 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:16.733699 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:16.734597 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.735301 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.736570 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.736948 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.737086 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.739043 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:16.739048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:16.739051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:16.739078 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:16.743066 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:16.743892 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:16.743899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:16.743905 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:16.744068 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:16.744458 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:16.744466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:16.744469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:16.744836 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:16.744993 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:16.746311 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0929 02:12:16.746561 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:12:16.756244 27537 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:44257
I20260929 02:12:16.756294 27671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:44257 every 8 connection(s)
I20260929 02:12:16.757437 27537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:12:16.759472 27672 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36893
I20260929 02:12:16.759651 27672 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:16.760205 27672 heartbeater.cc:511] Master 127.24.216.190:36893 requested a full tablet report, sending...
I20260929 02:12:16.761386 27187 ts_manager.cc:194] Registered new tserver with Master: d658ff37f51648c2ae3111d2e81aae83 (127.24.216.131:44257)
I20260929 02:12:16.762001 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 27537
I20260929 02:12:16.762221 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:12:16.762212 27187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:36589
I20260929 02:12:16.762699 25442 external_mini_cluster.cc:1428] Setting key 1262a41ce52e7d360a7cb45b686ba927
I20260929 02:12:16.765637 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:12:16.779587 27394 heartbeater.cc:503] Master 127.24.216.190:36893 was elected leader, sending a full tablet report...
Sep 29 02:12:16 dist-test-slave-9k0h krb5kdc[27142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:12:16 dist-test-slave-9k0h krb5kdc[27142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:16.818609 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.802382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.24.216.190:36893)
0929 02:12:16.802597 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:16.802606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:16.802838 (+ 232us) server_negotiation.cc:407] Connection header received
0929 02:12:16.803016 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:16.803024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:16.803120 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:16.803248 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:16.804524 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.805807 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.807068 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.807938 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.808085 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.811877 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:16.811887 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:16.811894 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:16.811935 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:16.815619 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:16.816372 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:16.816380 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:16.816385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:16.816535 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:16.816882 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:16.816890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:16.816895 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:16.817290 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:16.817551 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:16.817761 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 02:12:16.818035 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:12:16.823546 27187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:12:16.836740 27187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:12:16.873171 27686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.849752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:44257 (local address 127.0.0.1:36262)
0929 02:12:16.850549 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:16.850597 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:16.850763 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:16.863491 (+ 12728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:16.863502 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:16.863545 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:16.863984 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.863995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.866859 (+ 2864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.866868 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.869956 (+ 3088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.869969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.872217 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.872224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.872421 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.872466 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 02:12:16.872580 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 02:12:16.872753 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":17983,"threads_started":1}
I20260929 02:12:16.874111 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.850501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36262 (local address 127.24.216.131:44257)
0929 02:12:16.862702 (+ 12201us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:16.862714 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:16.862757 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:12:16.862890 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:16.862901 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:16.863211 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:16.863461 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:16.864304 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.866649 (+ 2345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.870138 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.871745 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.871921 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.872082 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:16.873655 (+ 1573us) server_negotiation.cc:299] Negotiation successful
0929 02:12:16.873815 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11988,"thread_start_us":11810,"threads_started":1}
I20260929 02:12:16.878405 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.851519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.24.216.130:38199)
0929 02:12:16.864704 (+ 13185us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:16.864714 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:16.869380 (+ 4666us) server_negotiation.cc:407] Connection header received
0929 02:12:16.870202 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:16.870210 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:16.870444 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:16.870611 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:16.871414 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.872725 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.875860 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.877112 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.877251 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.877400 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:16.877958 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 02:12:16.878127 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12995,"thread_start_us":16382,"threads_started":1}
I20260929 02:12:16.878062 27602 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3188ac07104469bc4d55dd2ed65a00 (DEFAULT_TABLE table=test-table [id=6212e821049a491ab6c028637b2aa245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:16.879257 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.851261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:38199 (local address 127.0.0.1:59298)
0929 02:12:16.869282 (+ 18021us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:16.869302 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:16.869967 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:16.870858 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:16.870863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:16.870886 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:16.871213 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.871222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.872905 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.872910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.875682 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.875694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.877596 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.877602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.877753 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.877799 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 02:12:16.878762 (+ 963us) client_negotiation.cc:240] Negotiation successful
0929 02:12:16.878898 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17895,"mutex_wait_us":97,"thread_start_us":17744,"threads_started":1}
I20260929 02:12:16.879601 27602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c3188ac07104469bc4d55dd2ed65a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:16.882594 27687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.850457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.24.216.129:41567)
0929 02:12:16.864248 (+ 13791us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:16.864259 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:16.867993 (+ 3734us) server_negotiation.cc:407] Connection header received
0929 02:12:16.868878 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:16.868888 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:16.869141 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:16.869325 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:16.870355 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.872201 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.877987 (+ 5786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.879806 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.879979 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.880183 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:16.881963 (+ 1780us) server_negotiation.cc:299] Negotiation successful
0929 02:12:16.882201 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13588,"thread_start_us":13464,"threads_started":1}
I20260929 02:12:16.883028 27463 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3188ac07104469bc4d55dd2ed65a00 (DEFAULT_TABLE table=test-table [id=6212e821049a491ab6c028637b2aa245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:16.883531 27685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.849841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:41567 (local address 127.0.0.1:52506)
0929 02:12:16.867337 (+ 17496us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:16.867906 (+ 569us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:16.868663 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:16.869451 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:16.869454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:16.869479 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:16.870088 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.870104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.873802 (+ 3698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.873813 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.877807 (+ 3994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.877823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.881483 (+ 3660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.881491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.881719 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.881766 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 02:12:16.882864 (+ 1098us) client_negotiation.cc:240] Negotiation successful
0929 02:12:16.883024 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17327,"mutex_wait_us":26,"thread_start_us":769,"threads_started":1}
I20260929 02:12:16.884299 27463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c3188ac07104469bc4d55dd2ed65a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:16.887428 27324 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3188ac07104469bc4d55dd2ed65a00 (DEFAULT_TABLE table=test-table [id=6212e821049a491ab6c028637b2aa245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:16.888806 27324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c3188ac07104469bc4d55dd2ed65a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:16.906888 27691 tablet_bootstrap.cc:492] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83: Bootstrap starting.
I20260929 02:12:16.911100 27691 tablet_bootstrap.cc:654] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:16.912591 27691 log.cc:824] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:16.916167 27692 tablet_bootstrap.cc:492] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc: Bootstrap starting.
I20260929 02:12:16.919966 27692 tablet_bootstrap.cc:654] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:16.920730 27693 tablet_bootstrap.cc:492] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df: Bootstrap starting.
I20260929 02:12:16.921238 27692 log.cc:824] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:16.921844 27691 tablet_bootstrap.cc:492] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83: No bootstrap required, opened a new log
I20260929 02:12:16.922124 27691 ts_tablet_manager.cc:1397] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260929 02:12:16.924037 27692 tablet_bootstrap.cc:492] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc: No bootstrap required, opened a new log
I20260929 02:12:16.924286 27692 ts_tablet_manager.cc:1397] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 02:12:16.924947 27693 tablet_bootstrap.cc:654] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:16.926517 27693 log.cc:824] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:16.932528 27691 raft_consensus.cc:348] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } }
I20260929 02:12:16.932976 27691 raft_consensus.cc:374] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:16.933079 27691 raft_consensus.cc:729] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d658ff37f51648c2ae3111d2e81aae83, State: Initialized, Role: FOLLOWER
I20260929 02:12:16.933758 27691 consensus_queue.cc:260] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } }
I20260929 02:12:16.935165 27672 heartbeater.cc:503] Master 127.24.216.190:36893 was elected leader, sending a full tablet report...
I20260929 02:12:16.935698 27692 raft_consensus.cc:348] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } }
I20260929 02:12:16.936105 27692 raft_consensus.cc:374] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:16.936213 27692 raft_consensus.cc:729] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f5c0bd7a67469bbd322302a0f410fc, State: Initialized, Role: FOLLOWER
I20260929 02:12:16.936595 27691 ts_tablet_manager.cc:1428] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260929 02:12:16.936853 27692 consensus_queue.cc:260] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } }
I20260929 02:12:16.939036 27693 tablet_bootstrap.cc:492] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df: No bootstrap required, opened a new log
I20260929 02:12:16.939311 27693 ts_tablet_manager.cc:1397] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.005s
I20260929 02:12:16.943233 27533 heartbeater.cc:503] Master 127.24.216.190:36893 was elected leader, sending a full tablet report...
I20260929 02:12:16.943866 27692 ts_tablet_manager.cc:1428] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc: Time spent starting tablet: real 0.019s user 0.014s sys 0.005s
I20260929 02:12:16.953222 27693 raft_consensus.cc:348] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } }
I20260929 02:12:16.953742 27693 raft_consensus.cc:374] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:16.953851 27693 raft_consensus.cc:729] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a7f958f73f466d9abacb5a240437df, State: Initialized, Role: FOLLOWER
I20260929 02:12:16.954538 27693 consensus_queue.cc:260] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } }
I20260929 02:12:16.956473 27693 ts_tablet_manager.cc:1428] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df: Time spent starting tablet: real 0.017s user 0.015s sys 0.002s
I20260929 02:12:16.975065 27698 raft_consensus.cc:482] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:12:16.975363 27698 raft_consensus.cc:504] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } }
I20260929 02:12:16.976918 27698 leader_election.cc:288] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d658ff37f51648c2ae3111d2e81aae83 (127.24.216.131:44257), b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567)
I20260929 02:12:16.985117 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.977748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:35941 (local address 127.24.216.131:44257)
0929 02:12:16.977970 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:16.977978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:16.978701 (+ 723us) server_negotiation.cc:407] Connection header received
0929 02:12:16.978934 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:16.978942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:16.979012 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:16.979070 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:16.980031 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.981124 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.983212 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.984204 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.984391 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.984465 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:16.984737 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 02:12:16.984864 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 02:12:16.985499 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.977252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:44257 (local address 127.24.216.130:35941)
0929 02:12:16.978594 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:16.978609 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:16.978768 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:16.979268 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:16.979272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:16.979315 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:16.979855 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.979868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.981298 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.981305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.983028 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.983039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.984418 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.984424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.984584 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.984607 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:12:16.984713 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 02:12:16.984830 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1238,"mutex_wait_us":1,"thread_start_us":759,"threads_started":1}
I20260929 02:12:16.986513 27627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3188ac07104469bc4d55dd2ed65a00" candidate_uuid: "20f5c0bd7a67469bbd322302a0f410fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d658ff37f51648c2ae3111d2e81aae83" is_pre_election: true
I20260929 02:12:16.987095 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.977372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:41567 (local address 127.24.216.130:40339)
0929 02:12:16.978707 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:16.978727 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:16.978925 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:16.979546 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:16.979553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:16.979588 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:16.980135 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.980149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.981521 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.981528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.984266 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:16.984282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.986272 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.986286 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:16.986473 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.986517 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 02:12:16.986619 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 02:12:16.986751 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"mutex_wait_us":58,"thread_start_us":938,"threads_started":1}
I20260929 02:12:16.987069 27627 raft_consensus.cc:2454] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f5c0bd7a67469bbd322302a0f410fc in term 0.
I20260929 02:12:16.987175 27687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:16.977356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:40339 (local address 127.24.216.129:41567)
0929 02:12:16.977565 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:16.977573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:16.978817 (+ 1244us) server_negotiation.cc:407] Connection header received
0929 02:12:16.979097 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:16.979105 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:16.979208 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:16.979272 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:16.980309 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.981367 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.984489 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:16.986037 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:16.986225 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:16.986326 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:16.986664 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 02:12:16.986851 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:12:16.987768 27415 leader_election.cc:302] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f5c0bd7a67469bbd322302a0f410fc, d658ff37f51648c2ae3111d2e81aae83; no voters:
I20260929 02:12:16.988164 27698 raft_consensus.cc:2781] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:12:16.988312 27698 raft_consensus.cc:482] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:12:16.988125 27344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3188ac07104469bc4d55dd2ed65a00" candidate_uuid: "20f5c0bd7a67469bbd322302a0f410fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a7f958f73f466d9abacb5a240437df" is_pre_election: true
I20260929 02:12:16.988409 27698 raft_consensus.cc:3037] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:16.988755 27344 raft_consensus.cc:2454] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f5c0bd7a67469bbd322302a0f410fc in term 0.
I20260929 02:12:16.990852 27698 raft_consensus.cc:504] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } }
I20260929 02:12:16.991500 27698 leader_election.cc:288] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [CANDIDATE]: Term 1 election: Requested vote from peers d658ff37f51648c2ae3111d2e81aae83 (127.24.216.131:44257), b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567)
I20260929 02:12:16.991813 27627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3188ac07104469bc4d55dd2ed65a00" candidate_uuid: "20f5c0bd7a67469bbd322302a0f410fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d658ff37f51648c2ae3111d2e81aae83"
I20260929 02:12:16.992007 27627 raft_consensus.cc:3037] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:16.992034 27344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3188ac07104469bc4d55dd2ed65a00" candidate_uuid: "20f5c0bd7a67469bbd322302a0f410fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a7f958f73f466d9abacb5a240437df"
I20260929 02:12:16.992215 27344 raft_consensus.cc:3037] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:16.995188 27627 raft_consensus.cc:2454] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f5c0bd7a67469bbd322302a0f410fc in term 1.
I20260929 02:12:16.995428 27344 raft_consensus.cc:2454] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f5c0bd7a67469bbd322302a0f410fc in term 1.
I20260929 02:12:16.995627 27415 leader_election.cc:302] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f5c0bd7a67469bbd322302a0f410fc, d658ff37f51648c2ae3111d2e81aae83; no voters:
I20260929 02:12:16.995910 27698 raft_consensus.cc:2781] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:16.996423 27698 raft_consensus.cc:686] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 LEADER]: Becoming Leader. State: Replica: 20f5c0bd7a67469bbd322302a0f410fc, State: Running, Role: LEADER
I20260929 02:12:16.996941 27698 consensus_queue.cc:237] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } }
I20260929 02:12:17.002404 27187 catalog_manager.cc:5495] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 20f5c0bd7a67469bbd322302a0f410fc (127.24.216.130). New cstate: current_term: 1 leader_uuid: "20f5c0bd7a67469bbd322302a0f410fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } health_report { overall_health: HEALTHY } } }
W20260929 02:12:17.008908 27673 tablet.cc:2369] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:12:17.011627 27534 tablet.cc:2369] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:12:17.017772 27395 tablet.cc:2369] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:12:17 dist-test-slave-9k0h krb5kdc[27142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647935, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:17.080018 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.064602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.24.216.190:36893)
0929 02:12:17.064822 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:17.064829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:17.065405 (+ 576us) server_negotiation.cc:407] Connection header received
0929 02:12:17.065538 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:17.065546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:17.065653 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:17.065786 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:17.067024 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.068350 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.069538 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.070458 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.070627 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.074362 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:17.074366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:17.074382 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:17.074408 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:17.076952 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:17.077988 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:17.077992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:17.077994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:17.078116 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:17.078549 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:17.078553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:17.078557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:17.078815 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:17.078957 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:17.079399 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 02:12:17.079701 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":63}
I20260929 02:12:17.080050 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.064624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36893 (local address 127.0.0.1:44180)
0929 02:12:17.065303 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:17.065319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:17.065439 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:17.065820 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:17.065823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:17.066597 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:17.066866 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:17.066874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.068477 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.068481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:17.069409 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:17.069419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.070648 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.070652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:17.070824 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.071546 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:17.071562 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:17.074140 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:17.077085 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:17.077102 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:17.077113 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:17.077869 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:17.078285 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:17.078292 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:17.078298 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:17.078441 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:17.078941 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:17.078971 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:17.079120 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 02:12:17.079301 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 02:12:17.079589 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":33,"thread_start_us":354,"threads_started":1}
I20260929 02:12:17.085364 27187 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:44180:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:12:17.086272 27187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:12:17.092092 27324 tablet_service.cc:1467] Processing CreateTablet for tablet d3803f09746e457dbdea1f45dcac6119 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a9554145c0b4848be94cf4bd4c3b824]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:12:17.092324 27602 tablet_service.cc:1467] Processing CreateTablet for tablet d3803f09746e457dbdea1f45dcac6119 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a9554145c0b4848be94cf4bd4c3b824]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:12:17.092199 27463 tablet_service.cc:1467] Processing CreateTablet for tablet d3803f09746e457dbdea1f45dcac6119 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a9554145c0b4848be94cf4bd4c3b824]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:12:17.092999 27463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3803f09746e457dbdea1f45dcac6119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:17.093029 27324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3803f09746e457dbdea1f45dcac6119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:17.093029 27602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3803f09746e457dbdea1f45dcac6119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:17.098909 27692 tablet_bootstrap.cc:492] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc: Bootstrap starting.
I20260929 02:12:17.100005 27691 tablet_bootstrap.cc:492] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83: Bootstrap starting.
I20260929 02:12:17.100628 27692 tablet_bootstrap.cc:654] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:17.101611 27691 tablet_bootstrap.cc:654] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:17.102368 27693 tablet_bootstrap.cc:492] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df: Bootstrap starting.
I20260929 02:12:17.103184 27692 tablet_bootstrap.cc:492] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc: No bootstrap required, opened a new log
I20260929 02:12:17.103335 27692 ts_tablet_manager.cc:1397] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260929 02:12:17.103935 27693 tablet_bootstrap.cc:654] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:17.104084 27692 raft_consensus.cc:348] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.104331 27692 raft_consensus.cc:374] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:17.104421 27692 raft_consensus.cc:729] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f5c0bd7a67469bbd322302a0f410fc, State: Initialized, Role: FOLLOWER
I20260929 02:12:17.104656 27692 consensus_queue.cc:260] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.105173 27698 tablet_replica.cc:439] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } }
I20260929 02:12:17.105506 27698 tablet_replica.cc:442] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:12:17.106968 27692 ts_tablet_manager.cc:1428] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:12:17.107340 27693 tablet_bootstrap.cc:492] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df: No bootstrap required, opened a new log
I20260929 02:12:17.107496 27693 ts_tablet_manager.cc:1397] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260929 02:12:17.108295 27691 tablet_bootstrap.cc:492] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83: No bootstrap required, opened a new log
I20260929 02:12:17.108263 27693 raft_consensus.cc:348] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.108426 27691 ts_tablet_manager.cc:1397] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 02:12:17.108505 27693 raft_consensus.cc:374] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:17.108603 27693 raft_consensus.cc:729] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a7f958f73f466d9abacb5a240437df, State: Initialized, Role: FOLLOWER
I20260929 02:12:17.108816 27693 consensus_queue.cc:260] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.109138 27691 raft_consensus.cc:348] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.109347 27693 ts_tablet_manager.cc:1428] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:12:17.109364 27691 raft_consensus.cc:374] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:17.109442 27691 raft_consensus.cc:729] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d658ff37f51648c2ae3111d2e81aae83, State: Initialized, Role: FOLLOWER
I20260929 02:12:17.109648 27691 consensus_queue.cc:260] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.109552 27699 tablet_replica.cc:439] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } }
I20260929 02:12:17.109926 27699 tablet_replica.cc:442] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:12:17.110278 27691 ts_tablet_manager.cc:1428] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 02:12:17.110129 27697 tablet_replica.cc:439] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } }
I20260929 02:12:17.110495 27697 tablet_replica.cc:442] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:12:17.348168 27697 raft_consensus.cc:482] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:12:17.348359 27697 raft_consensus.cc:504] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.349756 27697 leader_election.cc:288] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567), 20f5c0bd7a67469bbd322302a0f410fc (127.24.216.130:38199)
I20260929 02:12:17.356176 27687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.350226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:38333 (local address 127.24.216.129:41567)
0929 02:12:17.350386 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:17.350391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:17.350589 (+ 198us) server_negotiation.cc:407] Connection header received
0929 02:12:17.350767 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:17.350781 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:17.350870 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:17.350981 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:17.351478 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.352472 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.354228 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.355212 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.355335 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.355383 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:17.355765 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 02:12:17.355907 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:12:17.356202 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.350005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:41567 (local address 127.24.216.131:38333)
0929 02:12:17.350476 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:17.350496 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:17.350626 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:17.351099 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:17.351109 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:17.351133 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:17.351334 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:17.351341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.352605 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.352610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:17.354049 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:17.354061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.355364 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.355370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:17.355571 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.355605 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:12:17.355716 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 02:12:17.355937 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":275,"threads_started":1}
I20260929 02:12:17.356654 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.350623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:38199 (local address 127.24.216.131:60209)
0929 02:12:17.351193 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:17.351215 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:17.351356 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:17.351694 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:17.351698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:17.351722 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:17.351974 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:17.351984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.353148 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.353152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:17.354667 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:17.354680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.355887 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.355892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:17.356081 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.356109 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:12:17.356207 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 02:12:17.356350 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":293,"threads_started":1}
I20260929 02:12:17.356665 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.350759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:60209 (local address 127.24.216.130:38199)
0929 02:12:17.350989 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:17.351013 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:17.351304 (+ 291us) server_negotiation.cc:407] Connection header received
0929 02:12:17.351525 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:17.351529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:17.351594 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:17.351657 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:17.352113 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.353019 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.354832 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.355707 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.355836 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.355903 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:17.356265 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 02:12:17.356419 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:12:17.356967 27344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3803f09746e457dbdea1f45dcac6119" candidate_uuid: "d658ff37f51648c2ae3111d2e81aae83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a7f958f73f466d9abacb5a240437df" is_pre_election: true
I20260929 02:12:17.357167 27344 raft_consensus.cc:2454] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d658ff37f51648c2ae3111d2e81aae83 in term 0.
I20260929 02:12:17.357473 27483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3803f09746e457dbdea1f45dcac6119" candidate_uuid: "d658ff37f51648c2ae3111d2e81aae83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f5c0bd7a67469bbd322302a0f410fc" is_pre_election: true
I20260929 02:12:17.357623 27556 leader_election.cc:302] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2a7f958f73f466d9abacb5a240437df, d658ff37f51648c2ae3111d2e81aae83; no voters:
I20260929 02:12:17.357911 27483 raft_consensus.cc:2454] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d658ff37f51648c2ae3111d2e81aae83 in term 0.
I20260929 02:12:17.357997 27697 raft_consensus.cc:2781] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:12:17.358131 27697 raft_consensus.cc:482] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:12:17.358275 27697 raft_consensus.cc:3037] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:17.360352 27697 raft_consensus.cc:504] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.360868 27697 leader_election.cc:288] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [CANDIDATE]: Term 1 election: Requested vote from peers b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567), 20f5c0bd7a67469bbd322302a0f410fc (127.24.216.130:38199)
I20260929 02:12:17.361263 27344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3803f09746e457dbdea1f45dcac6119" candidate_uuid: "d658ff37f51648c2ae3111d2e81aae83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a7f958f73f466d9abacb5a240437df"
I20260929 02:12:17.361364 27483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3803f09746e457dbdea1f45dcac6119" candidate_uuid: "d658ff37f51648c2ae3111d2e81aae83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f5c0bd7a67469bbd322302a0f410fc"
I20260929 02:12:17.361479 27344 raft_consensus.cc:3037] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:17.361626 27483 raft_consensus.cc:3037] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:17.363544 27344 raft_consensus.cc:2454] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d658ff37f51648c2ae3111d2e81aae83 in term 1.
I20260929 02:12:17.363722 27483 raft_consensus.cc:2454] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d658ff37f51648c2ae3111d2e81aae83 in term 1.
I20260929 02:12:17.363900 27556 leader_election.cc:302] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2a7f958f73f466d9abacb5a240437df, d658ff37f51648c2ae3111d2e81aae83; no voters:
I20260929 02:12:17.364238 27697 raft_consensus.cc:2781] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:17.364657 27697 raft_consensus.cc:686] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [term 1 LEADER]: Becoming Leader. State: Replica: d658ff37f51648c2ae3111d2e81aae83, State: Running, Role: LEADER
I20260929 02:12:17.365109 27697 consensus_queue.cc:237] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.369027 27715 tablet_replica.cc:439] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83: TxnStatusTablet state changed. Reason: New leader d658ff37f51648c2ae3111d2e81aae83. Latest consensus state: current_term: 1 leader_uuid: "d658ff37f51648c2ae3111d2e81aae83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } }
I20260929 02:12:17.369377 27715 tablet_replica.cc:442] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:12:17.370539 27717 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:12:17.370829 27717 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:12:17.371374 27187 catalog_manager.cc:5495] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 reported cstate change: term changed from 0 to 1, leader changed from <none> to d658ff37f51648c2ae3111d2e81aae83 (127.24.216.131). New cstate: current_term: 1 leader_uuid: "d658ff37f51648c2ae3111d2e81aae83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } health_report { overall_health: HEALTHY } } }
I20260929 02:12:17.400418 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.394094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.24.216.131:44257)
0929 02:12:17.394284 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:17.394290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:17.394409 (+ 119us) server_negotiation.cc:407] Connection header received
0929 02:12:17.394675 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:17.394682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:17.394776 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:17.394903 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:12:17.395249 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.396222 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.398133 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.398624 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.398752 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.399339 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:12:17.399889 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:12:17.400002 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:17.400130 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 02:12:17.400195 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 02:12:17.400621 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.394072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:44257 (local address 127.0.0.1:36278)
0929 02:12:17.394322 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:17.394340 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:17.394498 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:17.394950 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:17.394956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:17.394968 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:12:17.395127 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:17.395134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.396362 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.396399 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:17.397969 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:17.397983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.398840 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.398850 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:17.399034 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.399111 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:12:17.400000 (+ 889us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:12:17.400020 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 02:12:17.400118 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 02:12:17.400262 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 02:12:17.408488 27483 raft_consensus.cc:1261] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Refusing update from remote peer d658ff37f51648c2ae3111d2e81aae83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:17.408587 27344 raft_consensus.cc:1261] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Refusing update from remote peer d658ff37f51648c2ae3111d2e81aae83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:17.409413 27697 consensus_queue.cc:1035] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:17.409771 27715 consensus_queue.cc:1035] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:17.419919 27699 tablet_replica.cc:439] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df: TxnStatusTablet state changed. Reason: New leader d658ff37f51648c2ae3111d2e81aae83. Latest consensus state: current_term: 1 leader_uuid: "d658ff37f51648c2ae3111d2e81aae83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } }
I20260929 02:12:17.420177 27699 tablet_replica.cc:442] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:12:17.422555 27698 tablet_replica.cc:439] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc: TxnStatusTablet state changed. Reason: New leader d658ff37f51648c2ae3111d2e81aae83. Latest consensus state: current_term: 1 leader_uuid: "d658ff37f51648c2ae3111d2e81aae83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } }
I20260929 02:12:17.422869 27698 tablet_replica.cc:442] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:12:17.422989 27720 mvcc.cc:205] Tried to move back new op lower bound from 7334493951616118784 to 7334493951451279360. Current Snapshot: MvccSnapshot[applied={T|T < 7334493951616118784}]
I20260929 02:12:17.423346 27697 tablet_replica.cc:439] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d658ff37f51648c2ae3111d2e81aae83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } }
I20260929 02:12:17.423623 27697 tablet_replica.cc:442] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:12:17.426692 27699 tablet_replica.cc:439] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d658ff37f51648c2ae3111d2e81aae83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } }
I20260929 02:12:17.426921 27699 tablet_replica.cc:442] T d3803f09746e457dbdea1f45dcac6119 P b2a7f958f73f466d9abacb5a240437df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:12:17.429198 27697 tablet_replica.cc:439] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d658ff37f51648c2ae3111d2e81aae83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } }
I20260929 02:12:17.429430 27697 tablet_replica.cc:442] T d3803f09746e457dbdea1f45dcac6119 P d658ff37f51648c2ae3111d2e81aae83: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:12:17.432221 27698 tablet_replica.cc:439] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d658ff37f51648c2ae3111d2e81aae83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } } }
I20260929 02:12:17.432457 27698 tablet_replica.cc:442] T d3803f09746e457dbdea1f45dcac6119 P 20f5c0bd7a67469bbd322302a0f410fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:12:17.444795 27698 consensus_queue.cc:1035] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:17.445125 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.439045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.24.216.130:38199)
0929 02:12:17.439196 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:17.439202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:17.439736 (+ 534us) server_negotiation.cc:407] Connection header received
0929 02:12:17.439962 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:17.439967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:17.440053 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:17.440190 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:12:17.440624 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.441318 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.442699 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.443108 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.443235 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.443702 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:12:17.444320 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:12:17.444457 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:17.444726 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 02:12:17.444807 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 02:12:17.451154 27698 consensus_queue.cc:1035] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:17.465888 27734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.460184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:44257 (local address 127.24.216.130:50773)
0929 02:12:17.460807 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:17.460822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:17.460931 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:17.461238 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:17.461241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:17.461249 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:12:17.461493 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:17.461500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.462873 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.462876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:17.463907 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:17.463916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.464899 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.464903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:17.465005 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.465050 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:12:17.465435 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:12:17.465442 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:12:17.465498 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 02:12:17.465580 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":342,"threads_started":1}
I20260929 02:12:17.465876 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.460301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:50773 (local address 127.24.216.131:44257)
0929 02:12:17.460518 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:17.460526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:17.460900 (+ 374us) server_negotiation.cc:407] Connection header received
0929 02:12:17.461041 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:17.461046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:17.461133 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:17.461232 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:12:17.461623 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.462757 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.464034 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.464732 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.464861 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.465216 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:12:17.465343 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:12:17.465431 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:17.465543 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 02:12:17.465614 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:12:17.506660 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.499957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:46879 (local address 127.24.216.130:38199)
0929 02:12:17.500147 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:17.500153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:17.500891 (+ 738us) server_negotiation.cc:407] Connection header received
0929 02:12:17.501389 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:17.501395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:17.501474 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:17.501564 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:12:17.502293 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.503077 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.505084 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.505453 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.505563 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.506061 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:12:17.506206 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:12:17.506294 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:17.506430 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 02:12:17.506487 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:12:17.506920 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.500051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:38199 (local address 127.24.216.131:46879)
0929 02:12:17.500785 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:17.500805 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:17.500942 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:17.501725 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:17.501730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:17.501742 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:12:17.502119 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:17.502130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.503199 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.503205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:17.504957 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:17.504972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.505648 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.505655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:17.505830 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.505902 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:12:17.506297 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:12:17.506308 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 02:12:17.506398 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 02:12:17.506533 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":302,"threads_started":1}
I20260929 02:12:17.537961 27186 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:17.538350 27186 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:17.542189 27186 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9: Sending DeleteTablet for 3 replicas of tablet 2c3188ac07104469bc4d55dd2ed65a00
I20260929 02:12:17.543203 27602 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c3188ac07104469bc4d55dd2ed65a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:17 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:36262
I20260929 02:12:17.543615 27324 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c3188ac07104469bc4d55dd2ed65a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:17 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:52506
I20260929 02:12:17.545504 27463 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c3188ac07104469bc4d55dd2ed65a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:17 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:59298
I20260929 02:12:17.550971 27186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:12:17.551612 27186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:12:17.553714 27743 tablet_replica.cc:330] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc: stopping tablet replica
I20260929 02:12:17.554349 27741 tablet_replica.cc:330] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83: stopping tablet replica
I20260929 02:12:17.554818 27743 raft_consensus.cc:2229] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:12:17.555294 27741 raft_consensus.cc:2229] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:17.555650 27742 tablet_replica.cc:330] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df: stopping tablet replica
I20260929 02:12:17.555984 27741 raft_consensus.cc:2258] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:17.556032 27742 raft_consensus.cc:2229] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:17.556365 27742 raft_consensus.cc:2258] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:17.556702 27743 raft_consensus.cc:2258] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:17.558354 27742 ts_tablet_manager.cc:1905] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:17.559530 27743 ts_tablet_manager.cc:1905] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:17.560422 27323 tablet_service.cc:1467] Processing CreateTablet for tablet ff8b8e0d679146c7beafaf71decbf3c5 (DEFAULT_TABLE table=test-table [id=4b7d1f8ccc214841ad1e63d7e4b0efc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:17.560943 27323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8b8e0d679146c7beafaf71decbf3c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:17.560922 27602 tablet_service.cc:1467] Processing CreateTablet for tablet ff8b8e0d679146c7beafaf71decbf3c5 (DEFAULT_TABLE table=test-table [id=4b7d1f8ccc214841ad1e63d7e4b0efc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:17.563323 27602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8b8e0d679146c7beafaf71decbf3c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:17.563591 27741 ts_tablet_manager.cc:1905] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:17.565846 27742 ts_tablet_manager.cc:1918] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:12:17.566000 27742 log.cc:1196] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/2c3188ac07104469bc4d55dd2ed65a00
I20260929 02:12:17.566555 27742 ts_tablet_manager.cc:1939] T 2c3188ac07104469bc4d55dd2ed65a00 P b2a7f958f73f466d9abacb5a240437df: Deleting consensus metadata
I20260929 02:12:17.567662 27693 tablet_bootstrap.cc:492] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df: Bootstrap starting.
I20260929 02:12:17.568225 27174 catalog_manager.cc:4841] TS b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567): tablet 2c3188ac07104469bc4d55dd2ed65a00 (table test-table [id=6212e821049a491ab6c028637b2aa245]) successfully deleted
I20260929 02:12:17.569132 27693 tablet_bootstrap.cc:654] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:17.571260 27693 tablet_bootstrap.cc:492] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df: No bootstrap required, opened a new log
I20260929 02:12:17.571422 27693 ts_tablet_manager.cc:1397] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 02:12:17.571966 27693 raft_consensus.cc:348] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.572161 27693 raft_consensus.cc:374] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:17.572237 27693 raft_consensus.cc:729] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a7f958f73f466d9abacb5a240437df, State: Initialized, Role: FOLLOWER
I20260929 02:12:17.572461 27693 consensus_queue.cc:260] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.573045 27693 ts_tablet_manager.cc:1428] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:17.560487 27463 tablet_service.cc:1467] Processing CreateTablet for tablet ff8b8e0d679146c7beafaf71decbf3c5 (DEFAULT_TABLE table=test-table [id=4b7d1f8ccc214841ad1e63d7e4b0efc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:17.582468 27691 tablet_bootstrap.cc:492] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83: Bootstrap starting.
I20260929 02:12:17.582719 27463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8b8e0d679146c7beafaf71decbf3c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:17.584111 27691 tablet_bootstrap.cc:654] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:17.590438 27741 ts_tablet_manager.cc:1918] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:12:17.590648 27741 log.cc:1196] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/2c3188ac07104469bc4d55dd2ed65a00
I20260929 02:12:17.591298 27741 ts_tablet_manager.cc:1939] T 2c3188ac07104469bc4d55dd2ed65a00 P d658ff37f51648c2ae3111d2e81aae83: Deleting consensus metadata
I20260929 02:12:17.591892 27743 ts_tablet_manager.cc:1918] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:12:17.591980 27691 tablet_bootstrap.cc:492] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83: No bootstrap required, opened a new log
I20260929 02:12:17.592145 27691 ts_tablet_manager.cc:1397] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260929 02:12:17.592904 27691 raft_consensus.cc:348] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.593075 27172 catalog_manager.cc:4841] TS d658ff37f51648c2ae3111d2e81aae83 (127.24.216.131:44257): tablet 2c3188ac07104469bc4d55dd2ed65a00 (table test-table [id=6212e821049a491ab6c028637b2aa245]) successfully deleted
I20260929 02:12:17.593150 27691 raft_consensus.cc:374] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:17.593223 27691 raft_consensus.cc:729] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d658ff37f51648c2ae3111d2e81aae83, State: Initialized, Role: FOLLOWER
I20260929 02:12:17.592981 27692 tablet_bootstrap.cc:492] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc: Bootstrap starting.
I20260929 02:12:17.593600 27743 log.cc:1196] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/2c3188ac07104469bc4d55dd2ed65a00
I20260929 02:12:17.593518 27691 consensus_queue.cc:260] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.595507 27692 tablet_bootstrap.cc:654] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:17.596311 27691 ts_tablet_manager.cc:1428] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 02:12:17.599005 27743 ts_tablet_manager.cc:1939] T 2c3188ac07104469bc4d55dd2ed65a00 P 20f5c0bd7a67469bbd322302a0f410fc: Deleting consensus metadata
I20260929 02:12:17.601265 27174 catalog_manager.cc:4841] TS 20f5c0bd7a67469bbd322302a0f410fc (127.24.216.130:38199): tablet 2c3188ac07104469bc4d55dd2ed65a00 (table test-table [id=6212e821049a491ab6c028637b2aa245]) successfully deleted
I20260929 02:12:17.607407 27692 tablet_bootstrap.cc:492] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc: No bootstrap required, opened a new log
I20260929 02:12:17.607594 27692 ts_tablet_manager.cc:1397] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20260929 02:12:17.608196 27692 raft_consensus.cc:348] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.608414 27692 raft_consensus.cc:374] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:17.608497 27692 raft_consensus.cc:729] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f5c0bd7a67469bbd322302a0f410fc, State: Initialized, Role: FOLLOWER
I20260929 02:12:17.608726 27692 consensus_queue.cc:260] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.609318 27692 ts_tablet_manager.cc:1428] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:12:17.768126 27715 raft_consensus.cc:482] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:12:17.768363 27715 raft_consensus.cc:504] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.769048 27715 leader_election.cc:288] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567), 20f5c0bd7a67469bbd322302a0f410fc (127.24.216.130:38199)
I20260929 02:12:17.770125 27344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8b8e0d679146c7beafaf71decbf3c5" candidate_uuid: "d658ff37f51648c2ae3111d2e81aae83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a7f958f73f466d9abacb5a240437df" is_pre_election: true
I20260929 02:12:17.770448 27344 raft_consensus.cc:2454] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d658ff37f51648c2ae3111d2e81aae83 in term 0.
I20260929 02:12:17.771103 27483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8b8e0d679146c7beafaf71decbf3c5" candidate_uuid: "d658ff37f51648c2ae3111d2e81aae83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f5c0bd7a67469bbd322302a0f410fc" is_pre_election: true
I20260929 02:12:17.771353 27483 raft_consensus.cc:2454] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d658ff37f51648c2ae3111d2e81aae83 in term 0.
I20260929 02:12:17.772228 27556 leader_election.cc:302] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f5c0bd7a67469bbd322302a0f410fc, d658ff37f51648c2ae3111d2e81aae83; no voters:
I20260929 02:12:17.772830 27715 raft_consensus.cc:2781] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:12:17.772931 27715 raft_consensus.cc:482] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:12:17.773038 27715 raft_consensus.cc:3037] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:17.776547 27715 raft_consensus.cc:504] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.777457 27344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8b8e0d679146c7beafaf71decbf3c5" candidate_uuid: "d658ff37f51648c2ae3111d2e81aae83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a7f958f73f466d9abacb5a240437df"
I20260929 02:12:17.777667 27344 raft_consensus.cc:3037] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:17.780774 27344 raft_consensus.cc:2454] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d658ff37f51648c2ae3111d2e81aae83 in term 1.
I20260929 02:12:17.781188 27556 leader_election.cc:302] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2a7f958f73f466d9abacb5a240437df, d658ff37f51648c2ae3111d2e81aae83; no voters:
I20260929 02:12:17.781481 27697 raft_consensus.cc:2781] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:17.781625 27697 raft_consensus.cc:686] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [term 1 LEADER]: Becoming Leader. State: Replica: d658ff37f51648c2ae3111d2e81aae83, State: Running, Role: LEADER
I20260929 02:12:17.781890 27697 consensus_queue.cc:237] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:17.782018 27483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8b8e0d679146c7beafaf71decbf3c5" candidate_uuid: "d658ff37f51648c2ae3111d2e81aae83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f5c0bd7a67469bbd322302a0f410fc"
I20260929 02:12:17.782243 27483 raft_consensus.cc:3037] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:17.782275 27715 leader_election.cc:288] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [CANDIDATE]: Term 1 election: Requested vote from peers b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567), 20f5c0bd7a67469bbd322302a0f410fc (127.24.216.130:38199)
I20260929 02:12:17.785413 27483 raft_consensus.cc:2454] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d658ff37f51648c2ae3111d2e81aae83 in term 1.
I20260929 02:12:17.801828 27186 catalog_manager.cc:5495] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 reported cstate change: term changed from 0 to 1, leader changed from <none> to d658ff37f51648c2ae3111d2e81aae83 (127.24.216.131). New cstate: current_term: 1 leader_uuid: "d658ff37f51648c2ae3111d2e81aae83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } health_report { overall_health: HEALTHY } } }
I20260929 02:12:17.827922 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.821513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.24.216.131:44257)
0929 02:12:17.821692 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:17.821698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:17.821828 (+ 130us) server_negotiation.cc:407] Connection header received
0929 02:12:17.821998 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:17.822003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:17.822082 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:17.822194 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:12:17.822633 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.823807 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.825720 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.826478 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.826609 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.827069 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:12:17.827243 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:12:17.827345 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:17.827521 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 02:12:17.827602 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:12:17.844394 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.838042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:58963 (local address 127.24.216.131:44257)
0929 02:12:17.838210 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:17.838215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:17.838233 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:12:17.838293 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:17.838298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:17.838393 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:17.838494 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:12:17.838927 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.840171 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.842235 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.842945 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.843074 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.843539 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:12:17.843660 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:12:17.843757 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:17.843926 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 02:12:17.844008 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:12:17.844415 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:17.837610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:44257 (local address 127.24.216.131:58963)
0929 02:12:17.837808 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:17.837826 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:17.837943 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:17.838568 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:17.838575 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:17.838595 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:12:17.838773 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:17.838783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.840327 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.840333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:17.842045 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:17.842058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:17.843098 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:17.843105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:17.843308 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:17.843361 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:12:17.843776 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:12:17.843786 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:12:17.843873 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 02:12:17.844020 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 02:12:17.855887 27344 raft_consensus.cc:1261] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Refusing update from remote peer d658ff37f51648c2ae3111d2e81aae83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:17.855877 27483 raft_consensus.cc:1261] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Refusing update from remote peer d658ff37f51648c2ae3111d2e81aae83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:17.856591 27715 consensus_queue.cc:1035] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:17.857154 27715 consensus_queue.cc:1035] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:17.874795 27722 mvcc.cc:205] Tried to move back new op lower bound from 7334493953451282432 to 7334493953156186112. Current Snapshot: MvccSnapshot[applied={T|T < 7334493953451282432}]
W20260929 02:12:17.895848 27720 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 02:12:19.485944 27187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:19.486186 27187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:19.489753 27187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9: Sending DeleteTablet for 3 replicas of tablet ff8b8e0d679146c7beafaf71decbf3c5
I20260929 02:12:19.490517 27323 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8b8e0d679146c7beafaf71decbf3c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:19 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:52506
I20260929 02:12:19.490582 27602 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8b8e0d679146c7beafaf71decbf3c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:19 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:36262
I20260929 02:12:19.490602 27463 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8b8e0d679146c7beafaf71decbf3c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:19 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:59298
I20260929 02:12:19.505966 27770 tablet_replica.cc:330] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83: stopping tablet replica
I20260929 02:12:19.505975 27771 tablet_replica.cc:330] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df: stopping tablet replica
I20260929 02:12:19.506197 27772 tablet_replica.cc:330] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc: stopping tablet replica
I20260929 02:12:19.506207 27770 raft_consensus.cc:2229] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:12:19.506207 27771 raft_consensus.cc:2229] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:19.506381 27772 raft_consensus.cc:2229] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:19.506445 27771 raft_consensus.cc:2258] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:19.506767 27770 raft_consensus.cc:2258] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:19.507018 27772 raft_consensus.cc:2258] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:19.507467 27771 ts_tablet_manager.cc:1905] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:19.507740 27770 ts_tablet_manager.cc:1905] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:19.508046 27772 ts_tablet_manager.cc:1905] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:19.514297 27771 ts_tablet_manager.cc:1918] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:12:19.514429 27771 log.cc:1196] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/ff8b8e0d679146c7beafaf71decbf3c5
I20260929 02:12:19.514742 27772 ts_tablet_manager.cc:1918] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:12:19.514880 27772 log.cc:1196] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/ff8b8e0d679146c7beafaf71decbf3c5
I20260929 02:12:19.514883 27771 ts_tablet_manager.cc:1939] T ff8b8e0d679146c7beafaf71decbf3c5 P b2a7f958f73f466d9abacb5a240437df: Deleting consensus metadata
I20260929 02:12:19.515017 27770 ts_tablet_manager.cc:1918] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:12:19.515172 27770 log.cc:1196] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/ff8b8e0d679146c7beafaf71decbf3c5
I20260929 02:12:19.515306 27772 ts_tablet_manager.cc:1939] T ff8b8e0d679146c7beafaf71decbf3c5 P 20f5c0bd7a67469bbd322302a0f410fc: Deleting consensus metadata
I20260929 02:12:19.515607 27770 ts_tablet_manager.cc:1939] T ff8b8e0d679146c7beafaf71decbf3c5 P d658ff37f51648c2ae3111d2e81aae83: Deleting consensus metadata
I20260929 02:12:19.516214 27174 catalog_manager.cc:4841] TS 20f5c0bd7a67469bbd322302a0f410fc (127.24.216.130:38199): tablet ff8b8e0d679146c7beafaf71decbf3c5 (table test-table [id=4b7d1f8ccc214841ad1e63d7e4b0efc0]) successfully deleted
I20260929 02:12:19.516503 27174 catalog_manager.cc:4841] TS b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567): tablet ff8b8e0d679146c7beafaf71decbf3c5 (table test-table [id=4b7d1f8ccc214841ad1e63d7e4b0efc0]) successfully deleted
I20260929 02:12:19.516714 27172 catalog_manager.cc:4841] TS d658ff37f51648c2ae3111d2e81aae83 (127.24.216.131:44257): tablet ff8b8e0d679146c7beafaf71decbf3c5 (table test-table [id=4b7d1f8ccc214841ad1e63d7e4b0efc0]) successfully deleted
Sep 29 02:12:19 dist-test-slave-9k0h krb5kdc[27142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:12:19 dist-test-slave-9k0h krb5kdc[27142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:19.550266 27780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:19.537024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.24.216.190:36893)
0929 02:12:19.537532 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:19.537536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:19.537557 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:12:19.537726 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:19.537730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:19.537793 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:19.537885 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:19.539294 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:19.540248 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:19.541433 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:19.541829 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:19.541917 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:19.545516 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:19.545520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:19.545523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:19.545552 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:19.548206 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:19.548764 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:19.548767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:19.548770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:19.548885 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:19.549153 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:19.549156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:19.549158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:19.549397 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:19.549555 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:19.549776 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 02:12:19.549965 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":278,"threads_started":1}
I20260929 02:12:19.553524 27187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:12:19.554117 27187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:12:19.560151 27323 tablet_service.cc:1467] Processing CreateTablet for tablet f6d2f7b2e308428abd6d1eb4c27fbaa7 (DEFAULT_TABLE table=test-table [id=311058684c58461faf2f59ce64aa8282]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:19.560264 27463 tablet_service.cc:1467] Processing CreateTablet for tablet f6d2f7b2e308428abd6d1eb4c27fbaa7 (DEFAULT_TABLE table=test-table [id=311058684c58461faf2f59ce64aa8282]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:19.560437 27602 tablet_service.cc:1467] Processing CreateTablet for tablet f6d2f7b2e308428abd6d1eb4c27fbaa7 (DEFAULT_TABLE table=test-table [id=311058684c58461faf2f59ce64aa8282]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:19.560611 27463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d2f7b2e308428abd6d1eb4c27fbaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:19.560609 27323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d2f7b2e308428abd6d1eb4c27fbaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:19.560823 27602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d2f7b2e308428abd6d1eb4c27fbaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:19.566661 27781 tablet_bootstrap.cc:492] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83: Bootstrap starting.
I20260929 02:12:19.567855 27782 tablet_bootstrap.cc:492] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc: Bootstrap starting.
I20260929 02:12:19.568382 27781 tablet_bootstrap.cc:654] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:19.568564 27783 tablet_bootstrap.cc:492] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df: Bootstrap starting.
I20260929 02:12:19.569901 27782 tablet_bootstrap.cc:654] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:19.570588 27783 tablet_bootstrap.cc:654] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:19.571820 27781 tablet_bootstrap.cc:492] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83: No bootstrap required, opened a new log
I20260929 02:12:19.571962 27781 ts_tablet_manager.cc:1397] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260929 02:12:19.572036 27782 tablet_bootstrap.cc:492] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc: No bootstrap required, opened a new log
I20260929 02:12:19.572166 27782 ts_tablet_manager.cc:1397] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:12:19.572610 27783 tablet_bootstrap.cc:492] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df: No bootstrap required, opened a new log
I20260929 02:12:19.572758 27783 ts_tablet_manager.cc:1397] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:12:19.572741 27781 raft_consensus.cc:348] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.572856 27782 raft_consensus.cc:348] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.572999 27781 raft_consensus.cc:374] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:19.573076 27782 raft_consensus.cc:374] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:19.573084 27781 raft_consensus.cc:729] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d658ff37f51648c2ae3111d2e81aae83, State: Initialized, Role: FOLLOWER
I20260929 02:12:19.573164 27782 raft_consensus.cc:729] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f5c0bd7a67469bbd322302a0f410fc, State: Initialized, Role: FOLLOWER
I20260929 02:12:19.573333 27781 consensus_queue.cc:260] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.573570 27783 raft_consensus.cc:348] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.573786 27783 raft_consensus.cc:374] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:19.573874 27781 ts_tablet_manager.cc:1428] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:12:19.573884 27783 raft_consensus.cc:729] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a7f958f73f466d9abacb5a240437df, State: Initialized, Role: FOLLOWER
I20260929 02:12:19.573487 27782 consensus_queue.cc:260] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.574120 27783 consensus_queue.cc:260] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.575410 27782 ts_tablet_manager.cc:1428] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:19.576177 27783 ts_tablet_manager.cc:1428] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260929 02:12:19.590194 27787 raft_consensus.cc:482] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:12:19.590407 27787 raft_consensus.cc:504] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.591084 27787 leader_election.cc:288] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567), d658ff37f51648c2ae3111d2e81aae83 (127.24.216.131:44257)
I20260929 02:12:19.591516 27627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d2f7b2e308428abd6d1eb4c27fbaa7" candidate_uuid: "20f5c0bd7a67469bbd322302a0f410fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d658ff37f51648c2ae3111d2e81aae83" is_pre_election: true
I20260929 02:12:19.591538 27343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d2f7b2e308428abd6d1eb4c27fbaa7" candidate_uuid: "20f5c0bd7a67469bbd322302a0f410fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a7f958f73f466d9abacb5a240437df" is_pre_election: true
I20260929 02:12:19.591750 27343 raft_consensus.cc:2454] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f5c0bd7a67469bbd322302a0f410fc in term 0.
I20260929 02:12:19.591790 27627 raft_consensus.cc:2454] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f5c0bd7a67469bbd322302a0f410fc in term 0.
I20260929 02:12:19.592221 27417 leader_election.cc:302] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f5c0bd7a67469bbd322302a0f410fc, b2a7f958f73f466d9abacb5a240437df; no voters:
I20260929 02:12:19.592499 27787 raft_consensus.cc:2781] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:12:19.592617 27787 raft_consensus.cc:482] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:12:19.592724 27787 raft_consensus.cc:3037] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:19.594759 27787 raft_consensus.cc:504] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.595244 27787 leader_election.cc:288] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [CANDIDATE]: Term 1 election: Requested vote from peers b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567), d658ff37f51648c2ae3111d2e81aae83 (127.24.216.131:44257)
I20260929 02:12:19.595641 27343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d2f7b2e308428abd6d1eb4c27fbaa7" candidate_uuid: "20f5c0bd7a67469bbd322302a0f410fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a7f958f73f466d9abacb5a240437df"
I20260929 02:12:19.595692 27627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d2f7b2e308428abd6d1eb4c27fbaa7" candidate_uuid: "20f5c0bd7a67469bbd322302a0f410fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d658ff37f51648c2ae3111d2e81aae83"
I20260929 02:12:19.595813 27343 raft_consensus.cc:3037] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:19.595963 27627 raft_consensus.cc:3037] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:19.597885 27343 raft_consensus.cc:2454] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f5c0bd7a67469bbd322302a0f410fc in term 1.
I20260929 02:12:19.598309 27417 leader_election.cc:302] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f5c0bd7a67469bbd322302a0f410fc, b2a7f958f73f466d9abacb5a240437df; no voters:
I20260929 02:12:19.598294 27627 raft_consensus.cc:2454] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f5c0bd7a67469bbd322302a0f410fc in term 1.
I20260929 02:12:19.598563 27787 raft_consensus.cc:2781] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:19.599015 27787 raft_consensus.cc:686] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 LEADER]: Becoming Leader. State: Replica: 20f5c0bd7a67469bbd322302a0f410fc, State: Running, Role: LEADER
I20260929 02:12:19.599354 27787 consensus_queue.cc:237] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.601723 27185 catalog_manager.cc:5495] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 20f5c0bd7a67469bbd322302a0f410fc (127.24.216.130). New cstate: current_term: 1 leader_uuid: "20f5c0bd7a67469bbd322302a0f410fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } health_report { overall_health: UNKNOWN } } }
I20260929 02:12:19.626941 27792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:19.621961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.24.216.130:38199)
0929 02:12:19.622557 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:19.622561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:19.622577 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:12:19.622620 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:19.622623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:19.622676 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:19.622776 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:12:19.623225 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:19.623958 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:19.625196 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:19.625609 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:19.625738 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:19.626227 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:12:19.626336 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:12:19.626435 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:19.626582 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 02:12:19.626650 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":369,"threads_started":1}
I20260929 02:12:19.635533 27627 raft_consensus.cc:1261] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Refusing update from remote peer 20f5c0bd7a67469bbd322302a0f410fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:19.635622 27343 raft_consensus.cc:1261] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Refusing update from remote peer 20f5c0bd7a67469bbd322302a0f410fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:19.636072 27787 consensus_queue.cc:1035] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:19.636382 27789 consensus_queue.cc:1035] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:19.640200 27754 mvcc.cc:205] Tried to move back new op lower bound from 7334493960742301696 to 7334493960601276416. Current Snapshot: MvccSnapshot[applied={T|T < 7334493960742301696}]
I20260929 02:12:19.679744 27186 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:19.679976 27186 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:19.682574 27186 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9: Sending DeleteTablet for 3 replicas of tablet f6d2f7b2e308428abd6d1eb4c27fbaa7
I20260929 02:12:19.683239 27323 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d2f7b2e308428abd6d1eb4c27fbaa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:19 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:52506
I20260929 02:12:19.683355 27463 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d2f7b2e308428abd6d1eb4c27fbaa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:19 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:59298
I20260929 02:12:19.683380 27602 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d2f7b2e308428abd6d1eb4c27fbaa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:19 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:36262
I20260929 02:12:19.683506 27771 tablet_replica.cc:330] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df: stopping tablet replica
I20260929 02:12:19.683589 27772 tablet_replica.cc:330] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc: stopping tablet replica
I20260929 02:12:19.683620 27770 tablet_replica.cc:330] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83: stopping tablet replica
I20260929 02:12:19.683699 27771 raft_consensus.cc:2229] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:19.683784 27772 raft_consensus.cc:2229] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:12:19.683806 27770 raft_consensus.cc:2229] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:19.683866 27771 raft_consensus.cc:2258] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:19.683974 27770 raft_consensus.cc:2258] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:19.684046 27772 raft_consensus.cc:2258] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:19.685115 27772 ts_tablet_manager.cc:1905] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:19.685156 27770 ts_tablet_manager.cc:1905] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:19.684960 27186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 02:12:19.685407 27771 ts_tablet_manager.cc:1905] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 02:12:19.685582 27186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:12:19.691774 27771 ts_tablet_manager.cc:1918] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:12:19.691897 27770 ts_tablet_manager.cc:1918] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:12:19.691927 27771 log.cc:1196] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/f6d2f7b2e308428abd6d1eb4c27fbaa7
I20260929 02:12:19.692035 27770 log.cc:1196] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/f6d2f7b2e308428abd6d1eb4c27fbaa7
I20260929 02:12:19.692075 27772 ts_tablet_manager.cc:1918] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:12:19.692200 27772 log.cc:1196] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/f6d2f7b2e308428abd6d1eb4c27fbaa7
I20260929 02:12:19.692399 27771 ts_tablet_manager.cc:1939] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P b2a7f958f73f466d9abacb5a240437df: Deleting consensus metadata
I20260929 02:12:19.692600 27772 ts_tablet_manager.cc:1939] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P 20f5c0bd7a67469bbd322302a0f410fc: Deleting consensus metadata
I20260929 02:12:19.693485 27174 catalog_manager.cc:4841] TS b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567): tablet f6d2f7b2e308428abd6d1eb4c27fbaa7 (table test-table [id=311058684c58461faf2f59ce64aa8282]) successfully deleted
I20260929 02:12:19.693491 27323 tablet_service.cc:1467] Processing CreateTablet for tablet 377bac2b420741cfa464993a2c0c9927 (DEFAULT_TABLE table=test-table [id=8cbd5c9ca8ce492182d8828baff869ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:19.692862 27602 tablet_service.cc:1467] Processing CreateTablet for tablet 377bac2b420741cfa464993a2c0c9927 (DEFAULT_TABLE table=test-table [id=8cbd5c9ca8ce492182d8828baff869ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:19.693908 27323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377bac2b420741cfa464993a2c0c9927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:19.694095 27602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377bac2b420741cfa464993a2c0c9927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:19.694624 27770 ts_tablet_manager.cc:1939] T f6d2f7b2e308428abd6d1eb4c27fbaa7 P d658ff37f51648c2ae3111d2e81aae83: Deleting consensus metadata
I20260929 02:12:19.695546 27172 catalog_manager.cc:4841] TS d658ff37f51648c2ae3111d2e81aae83 (127.24.216.131:44257): tablet f6d2f7b2e308428abd6d1eb4c27fbaa7 (table test-table [id=311058684c58461faf2f59ce64aa8282]) successfully deleted
I20260929 02:12:19.698925 27783 tablet_bootstrap.cc:492] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df: Bootstrap starting.
I20260929 02:12:19.699069 27781 tablet_bootstrap.cc:492] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83: Bootstrap starting.
I20260929 02:12:19.699075 27174 catalog_manager.cc:4841] TS 20f5c0bd7a67469bbd322302a0f410fc (127.24.216.130:38199): tablet f6d2f7b2e308428abd6d1eb4c27fbaa7 (table test-table [id=311058684c58461faf2f59ce64aa8282]) successfully deleted
I20260929 02:12:19.699749 27463 tablet_service.cc:1467] Processing CreateTablet for tablet 377bac2b420741cfa464993a2c0c9927 (DEFAULT_TABLE table=test-table [id=8cbd5c9ca8ce492182d8828baff869ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:19.700129 27463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377bac2b420741cfa464993a2c0c9927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:19.700469 27783 tablet_bootstrap.cc:654] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:19.700719 27781 tablet_bootstrap.cc:654] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:19.702473 27783 tablet_bootstrap.cc:492] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df: No bootstrap required, opened a new log
I20260929 02:12:19.702620 27783 ts_tablet_manager.cc:1397] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 02:12:19.703284 27783 raft_consensus.cc:348] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.703524 27783 raft_consensus.cc:374] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:19.703612 27783 raft_consensus.cc:729] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a7f958f73f466d9abacb5a240437df, State: Initialized, Role: FOLLOWER
I20260929 02:12:19.703848 27783 consensus_queue.cc:260] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.704345 27783 ts_tablet_manager.cc:1428] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:19.705529 27781 tablet_bootstrap.cc:492] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83: No bootstrap required, opened a new log
I20260929 02:12:19.705720 27781 ts_tablet_manager.cc:1397] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20260929 02:12:19.706640 27782 tablet_bootstrap.cc:492] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc: Bootstrap starting.
I20260929 02:12:19.706933 27781 raft_consensus.cc:348] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.707156 27781 raft_consensus.cc:374] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:19.707232 27781 raft_consensus.cc:729] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d658ff37f51648c2ae3111d2e81aae83, State: Initialized, Role: FOLLOWER
I20260929 02:12:19.707475 27781 consensus_queue.cc:260] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.707964 27781 ts_tablet_manager.cc:1428] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:12:19.708225 27782 tablet_bootstrap.cc:654] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:19.710314 27782 tablet_bootstrap.cc:492] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc: No bootstrap required, opened a new log
I20260929 02:12:19.710472 27782 ts_tablet_manager.cc:1397] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:12:19.711064 27782 raft_consensus.cc:348] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.711267 27782 raft_consensus.cc:374] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:19.711359 27782 raft_consensus.cc:729] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f5c0bd7a67469bbd322302a0f410fc, State: Initialized, Role: FOLLOWER
I20260929 02:12:19.711599 27782 consensus_queue.cc:260] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.712538 27782 ts_tablet_manager.cc:1428] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:19.894443 27797 raft_consensus.cc:482] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:12:19.894680 27797 raft_consensus.cc:504] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.895335 27797 leader_election.cc:288] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567), d658ff37f51648c2ae3111d2e81aae83 (127.24.216.131:44257)
I20260929 02:12:19.895830 27627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377bac2b420741cfa464993a2c0c9927" candidate_uuid: "20f5c0bd7a67469bbd322302a0f410fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d658ff37f51648c2ae3111d2e81aae83" is_pre_election: true
I20260929 02:12:19.895867 27344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377bac2b420741cfa464993a2c0c9927" candidate_uuid: "20f5c0bd7a67469bbd322302a0f410fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a7f958f73f466d9abacb5a240437df" is_pre_election: true
I20260929 02:12:19.896091 27627 raft_consensus.cc:2454] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f5c0bd7a67469bbd322302a0f410fc in term 0.
I20260929 02:12:19.896214 27344 raft_consensus.cc:2454] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f5c0bd7a67469bbd322302a0f410fc in term 0.
I20260929 02:12:19.896569 27415 leader_election.cc:302] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f5c0bd7a67469bbd322302a0f410fc, d658ff37f51648c2ae3111d2e81aae83; no voters:
I20260929 02:12:19.896880 27797 raft_consensus.cc:2781] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:12:19.897013 27797 raft_consensus.cc:482] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:12:19.897099 27797 raft_consensus.cc:3037] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:19.899192 27797 raft_consensus.cc:504] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.899672 27797 leader_election.cc:288] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [CANDIDATE]: Term 1 election: Requested vote from peers b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567), d658ff37f51648c2ae3111d2e81aae83 (127.24.216.131:44257)
I20260929 02:12:19.900064 27344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377bac2b420741cfa464993a2c0c9927" candidate_uuid: "20f5c0bd7a67469bbd322302a0f410fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a7f958f73f466d9abacb5a240437df"
I20260929 02:12:19.900133 27627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377bac2b420741cfa464993a2c0c9927" candidate_uuid: "20f5c0bd7a67469bbd322302a0f410fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d658ff37f51648c2ae3111d2e81aae83"
I20260929 02:12:19.900292 27344 raft_consensus.cc:3037] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:19.900310 27627 raft_consensus.cc:3037] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:19.903105 27344 raft_consensus.cc:2454] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f5c0bd7a67469bbd322302a0f410fc in term 1.
I20260929 02:12:19.903129 27627 raft_consensus.cc:2454] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f5c0bd7a67469bbd322302a0f410fc in term 1.
I20260929 02:12:19.903478 27415 leader_election.cc:302] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20f5c0bd7a67469bbd322302a0f410fc, b2a7f958f73f466d9abacb5a240437df, d658ff37f51648c2ae3111d2e81aae83; no voters:
I20260929 02:12:19.903724 27797 raft_consensus.cc:2781] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:19.903873 27797 raft_consensus.cc:686] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 LEADER]: Becoming Leader. State: Replica: 20f5c0bd7a67469bbd322302a0f410fc, State: Running, Role: LEADER
I20260929 02:12:19.904152 27797 consensus_queue.cc:237] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } }
I20260929 02:12:19.906598 27186 catalog_manager.cc:5495] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 20f5c0bd7a67469bbd322302a0f410fc (127.24.216.130). New cstate: current_term: 1 leader_uuid: "20f5c0bd7a67469bbd322302a0f410fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f5c0bd7a67469bbd322302a0f410fc" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 } health_report { overall_health: UNKNOWN } } }
I20260929 02:12:19.955060 27344 raft_consensus.cc:1261] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Refusing update from remote peer 20f5c0bd7a67469bbd322302a0f410fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:19.955305 27627 raft_consensus.cc:1261] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Refusing update from remote peer 20f5c0bd7a67469bbd322302a0f410fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:19.955556 27797 consensus_queue.cc:1035] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a7f958f73f466d9abacb5a240437df" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:19.955853 27798 consensus_queue.cc:1035] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d658ff37f51648c2ae3111d2e81aae83" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 44257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:12:19.973519 27754 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 02:12:21.546330 27185 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:21.546578 27185 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:21.550344 27185 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37b5d010f8514d60adcbd12e7f4321f9: Sending DeleteTablet for 3 replicas of tablet 377bac2b420741cfa464993a2c0c9927
I20260929 02:12:21.551049 27463 tablet_service.cc:1514] Processing DeleteTablet for tablet 377bac2b420741cfa464993a2c0c9927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:21 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:59298
I20260929 02:12:21.551050 27323 tablet_service.cc:1514] Processing DeleteTablet for tablet 377bac2b420741cfa464993a2c0c9927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:21 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:52506
I20260929 02:12:21.551131 27602 tablet_service.cc:1514] Processing DeleteTablet for tablet 377bac2b420741cfa464993a2c0c9927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:21 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:36262
I20260929 02:12:21.551681 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 27258
I20260929 02:12:21.561995 27829 tablet_replica.cc:330] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83: stopping tablet replica
I20260929 02:12:21.562263 27829 raft_consensus.cc:2229] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:21.562527 27829 raft_consensus.cc:2258] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:21.564061 27829 ts_tablet_manager.cc:1905] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:21.571301 27829 ts_tablet_manager.cc:1918] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:12:21.571448 27829 log.cc:1196] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/377bac2b420741cfa464993a2c0c9927
I20260929 02:12:21.571921 27829 ts_tablet_manager.cc:1939] T 377bac2b420741cfa464993a2c0c9927 P d658ff37f51648c2ae3111d2e81aae83: Deleting consensus metadata
I20260929 02:12:21.573074 27172 catalog_manager.cc:4841] TS d658ff37f51648c2ae3111d2e81aae83 (127.24.216.131:44257): tablet 377bac2b420741cfa464993a2c0c9927 (table test-table [id=8cbd5c9ca8ce492182d8828baff869ff]) successfully deleted
I20260929 02:12:21.573726 27827 tablet_replica.cc:330] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df: stopping tablet replica
I20260929 02:12:21.573961 27828 tablet_replica.cc:330] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc: stopping tablet replica
I20260929 02:12:21.573962 27827 raft_consensus.cc:2229] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:21.574230 27827 raft_consensus.cc:2258] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:21.574220 27828 raft_consensus.cc:2229] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:12:21.574615 27828 raft_consensus.cc:2258] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 02:12:21 dist-test-slave-9k0h krb5kdc[27142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:12:21.575145 27827 ts_tablet_manager.cc:1905] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:21.575572 27828 ts_tablet_manager.cc:1905] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:21.580680 27827 ts_tablet_manager.cc:1918] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:12:21.580832 27827 log.cc:1196] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/377bac2b420741cfa464993a2c0c9927
I20260929 02:12:21.581285 27827 ts_tablet_manager.cc:1939] T 377bac2b420741cfa464993a2c0c9927 P b2a7f958f73f466d9abacb5a240437df: Deleting consensus metadata
I20260929 02:12:21.582391 27174 catalog_manager.cc:4841] TS b2a7f958f73f466d9abacb5a240437df (127.24.216.129:41567): tablet 377bac2b420741cfa464993a2c0c9927 (table test-table [id=8cbd5c9ca8ce492182d8828baff869ff]) successfully deleted
I20260929 02:12:21.582857 27828 ts_tablet_manager.cc:1918] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:12:21.582995 27828 log.cc:1196] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/377bac2b420741cfa464993a2c0c9927
I20260929 02:12:21.583531 27828 ts_tablet_manager.cc:1939] T 377bac2b420741cfa464993a2c0c9927 P 20f5c0bd7a67469bbd322302a0f410fc: Deleting consensus metadata
I20260929 02:12:21.584607 27174 catalog_manager.cc:4841] TS 20f5c0bd7a67469bbd322302a0f410fc (127.24.216.130:38199): tablet 377bac2b420741cfa464993a2c0c9927 (table test-table [id=8cbd5c9ca8ce492182d8828baff869ff]) successfully deleted
I20260929 02:12:21.584892 27831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:21.553778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.24.216.129:41567)
0929 02:12:21.563353 (+ 9575us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:21.563360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:21.563778 (+ 418us) server_negotiation.cc:407] Connection header received
0929 02:12:21.563837 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:21.563842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:21.563917 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:21.564013 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:21.566009 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:21.567183 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:21.569466 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:21.570117 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:21.570243 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:21.576045 (+ 5802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:21.576066 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:21.576092 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:21.576140 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:21.580527 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:21.581445 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:21.581455 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:21.581459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:21.581648 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:21.582442 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:21.582447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:21.582450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:21.582907 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:21.583206 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:21.584407 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0929 02:12:21.584594 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9385,"thread_start_us":9662,"threads_started":1}
W20260929 02:12:21.585479 27389 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52516
W20260929 02:12:21.586041 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 27258: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:12:21.586289 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 27258
I20260929 02:12:21.604645 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 27398
Sep 29 02:12:21 dist-test-slave-9k0h krb5kdc[27142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:12:21.619127 27832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:21.605371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.24.216.130:38199)
0929 02:12:21.605918 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:21.605922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:21.605938 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:12:21.605985 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:21.605989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:21.606049 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:21.606138 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:21.607283 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:21.607996 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:21.609146 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:21.609541 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:21.609626 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:21.613699 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:21.613711 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:21.613726 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:21.613755 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:21.616683 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:21.617233 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:21.617239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:21.617242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:21.617361 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:21.617663 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:21.617667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:21.617669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:21.618004 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:21.618358 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:21.618578 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 02:12:21.618726 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":315,"threads_started":1}
W20260929 02:12:21.619745 27528 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59338
W20260929 02:12:21.620370 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 27398: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:12:21.620510 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 27398
I20260929 02:12:21.643519 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 27537
Sep 29 02:12:21 dist-test-slave-9k0h krb5kdc[27142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:12:21.657923 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:21.644225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.24.216.131:44257)
0929 02:12:21.644750 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:21.644754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:21.644770 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:12:21.644828 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:21.644832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:21.644891 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:21.644982 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:21.646260 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:21.647001 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:21.648120 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:21.648516 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:21.648610 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:21.652792 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:21.652806 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:21.652823 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:21.652857 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:21.655586 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:21.656161 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:21.656168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:21.656170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:21.656301 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:21.656613 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:21.656622 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:21.656624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:21.656960 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:21.657190 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:21.657402 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 02:12:21.657550 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":286,"threads_started":1}
W20260929 02:12:21.658717 27667 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36290
W20260929 02:12:21.659327 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 27537: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:12:21.659499 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 27537
I20260929 02:12:21.684314 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 27155
I20260929 02:12:21.685475 27227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:21.799950 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 27155
2026-09-29T02:12:21Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8286 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:12:21 dist-test-slave-9k0h krb5kdc[27837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:12:21 dist-test-slave-9k0h krb5kdc[27837](info): set up 2 sockets
Sep 29 02:12:21 dist-test-slave-9k0h krb5kdc[27837](info): commencing operation
krb5kdc: starting...
W20260929 02:12:22.878479 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:12:22 dist-test-slave-9k0h krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:12:23.059787 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:39961
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:39961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:23.371485 27850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:23.371774 27850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:23.371856 27850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:23.378875 27850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:23.378968 27850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:23.379011 27850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:23.379067 27850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:23.379103 27850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:12:23.388314 27850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:39961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:39961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:23.389549 27850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:23.390878 27850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:23.402482 27858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:23.402565 27856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:23.402523 27855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:23.403676 27850 server_base.cc:1034] running on GCE node
I20260929 02:12:23.404460 27850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:23.405848 27850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:23.407078 27850 hybrid_clock.cc:648] HybridClock initialized: now 1790647943407021 us; error 46 us; skew 500 ppm
I20260929 02:12:23.407409 27850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:23 dist-test-slave-9k0h krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647943, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:23.410436 27850 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:12:23.411846 27850 webserver.cc:460] Webserver started at http://127.24.216.190:44471/ using document root <none> and password file <none>
I20260929 02:12:23.412503 27850 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:23.412642 27850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:23.412947 27850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:23.415827 27850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "f92dc4d8f35b4f54aeda9ff043b9ada1"
format_stamp: "Formatted at 2026-09-29 02:12:23 on dist-test-slave-9k0h"
server_key: "ca915377c6cc68574e1c37484aaeb109"
server_key_iv: "738a91a6c188001cb4d6ff28c8f7f049"
server_key_version: "encryptionkey@0"
I20260929 02:12:23.416570 27850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "f92dc4d8f35b4f54aeda9ff043b9ada1"
format_stamp: "Formatted at 2026-09-29 02:12:23 on dist-test-slave-9k0h"
server_key: "ca915377c6cc68574e1c37484aaeb109"
server_key_iv: "738a91a6c188001cb4d6ff28c8f7f049"
server_key_version: "encryptionkey@0"
I20260929 02:12:23.421423 27850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:12:23.424492 27865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:23.425734 27850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:23.425890 27850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "f92dc4d8f35b4f54aeda9ff043b9ada1"
format_stamp: "Formatted at 2026-09-29 02:12:23 on dist-test-slave-9k0h"
server_key: "ca915377c6cc68574e1c37484aaeb109"
server_key_iv: "738a91a6c188001cb4d6ff28c8f7f049"
server_key_version: "encryptionkey@0"
I20260929 02:12:23.426070 27850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:23.454994 27850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:23.458703 27850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:23.458998 27850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:23.476311 27850 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:39961
I20260929 02:12:23.476333 27916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:39961 every 8 connection(s)
I20260929 02:12:23.477602 27850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:23.481096 27917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:23.486027 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 27850
I20260929 02:12:23.486274 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:23.486897 25442 external_mini_cluster.cc:1428] Setting key e0bb795dece6427d64361d6260849b23
I20260929 02:12:23.492581 27917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1: Bootstrap starting.
I20260929 02:12:23.496129 27917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:23.497148 27917 log.cc:824] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:23.499840 27917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1: No bootstrap required, opened a new log
Sep 29 02:12:23 dist-test-slave-9k0h krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:23.509588 27917 raft_consensus.cc:348] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92dc4d8f35b4f54aeda9ff043b9ada1" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39961 } }
I20260929 02:12:23.509984 27917 raft_consensus.cc:374] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:23.510087 27917 raft_consensus.cc:729] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92dc4d8f35b4f54aeda9ff043b9ada1, State: Initialized, Role: FOLLOWER
I20260929 02:12:23.510257 27920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:23.488841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.24.216.190:39961)
0929 02:12:23.489772 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:23.489796 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:23.489852 (+ 56us) server_negotiation.cc:407] Connection header received
0929 02:12:23.492171 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:23.492197 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:23.492646 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:23.493058 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:23.494307 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:23.496225 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:23.497572 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:23.498450 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:23.498633 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:23.502242 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:23.502252 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:23.502274 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:23.502313 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:23.506125 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:23.506721 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:23.506729 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:23.506737 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:23.506928 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:23.507267 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:23.507272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:23.507275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:23.507997 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:23.508275 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:23.508574 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 02:12:23.508972 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":447,"threads_started":1}
I20260929 02:12:23.510741 27917 consensus_queue.cc:260] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92dc4d8f35b4f54aeda9ff043b9ada1" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39961 } }
I20260929 02:12:23.510989 27917 raft_consensus.cc:388] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:23.511121 27917 raft_consensus.cc:482] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:23.511300 27917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:23.513541 27917 raft_consensus.cc:504] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92dc4d8f35b4f54aeda9ff043b9ada1" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39961 } }
I20260929 02:12:23.513983 27917 leader_election.cc:302] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f92dc4d8f35b4f54aeda9ff043b9ada1; no voters:
I20260929 02:12:23.514549 27917 leader_election.cc:288] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:23.514684 27922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:23.515350 27922 raft_consensus.cc:686] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [term 1 LEADER]: Becoming Leader. State: Replica: f92dc4d8f35b4f54aeda9ff043b9ada1, State: Running, Role: LEADER
I20260929 02:12:23.515811 27922 consensus_queue.cc:237] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92dc4d8f35b4f54aeda9ff043b9ada1" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39961 } }
I20260929 02:12:23.516496 27917 sys_catalog.cc:564] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:23.518622 27923 sys_catalog.cc:455] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f92dc4d8f35b4f54aeda9ff043b9ada1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92dc4d8f35b4f54aeda9ff043b9ada1" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39961 } } }
I20260929 02:12:23.518965 27923 sys_catalog.cc:458] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:23.519469 27924 sys_catalog.cc:455] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f92dc4d8f35b4f54aeda9ff043b9ada1. Latest consensus state: current_term: 1 leader_uuid: "f92dc4d8f35b4f54aeda9ff043b9ada1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92dc4d8f35b4f54aeda9ff043b9ada1" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39961 } } }
I20260929 02:12:23.519740 27924 sys_catalog.cc:458] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:23.521571 27928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:23.526602 27928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:23.534375 27928 catalog_manager.cc:1347] Generated new cluster ID: fae92f5343254d378c4b7dccf67aa230
I20260929 02:12:23.534473 27928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:23.566819 27928 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:23.568674 27928 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:23.585891 27928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f92dc4d8f35b4f54aeda9ff043b9ada1: Generated new TSK 0
I20260929 02:12:23.586747 27928 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:12:23.641355 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39961
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:23.963184 27943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:23.963477 27943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:23.963629 27943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:23.970633 27943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:23.970778 27943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:23.970980 27943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:12:23.980365 27943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:23.981657 27943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:23.983095 27943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:23.995923 27949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:23.995905 27948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:23.996122 27951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:23.996418 27943 server_base.cc:1034] running on GCE node
I20260929 02:12:23.997129 27943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:23.998020 27943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:23.999240 27943 hybrid_clock.cc:648] HybridClock initialized: now 1790647943999198 us; error 58 us; skew 500 ppm
I20260929 02:12:23.999565 27943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:23 dist-test-slave-9k0h krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647943, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:24.002532 27943 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:12:24.003868 27943 webserver.cc:460] Webserver started at http://127.24.216.129:46075/ using document root <none> and password file <none>
I20260929 02:12:24.004519 27943 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:24.004680 27943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:24.005002 27943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:24.007797 27943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "8a45ecdad58245bf847fdd0f39ebdfc7"
format_stamp: "Formatted at 2026-09-29 02:12:23 on dist-test-slave-9k0h"
server_key: "3f4604c125b898136306ec1f17915086"
server_key_iv: "91f981ea65e5236443d13ef08cdbc5b3"
server_key_version: "encryptionkey@0"
I20260929 02:12:24.008627 27943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "8a45ecdad58245bf847fdd0f39ebdfc7"
format_stamp: "Formatted at 2026-09-29 02:12:23 on dist-test-slave-9k0h"
server_key: "3f4604c125b898136306ec1f17915086"
server_key_iv: "91f981ea65e5236443d13ef08cdbc5b3"
server_key_version: "encryptionkey@0"
I20260929 02:12:24.013707 27943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:12:24.017019 27958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:24.018452 27943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:24.018620 27943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "8a45ecdad58245bf847fdd0f39ebdfc7"
format_stamp: "Formatted at 2026-09-29 02:12:23 on dist-test-slave-9k0h"
server_key: "3f4604c125b898136306ec1f17915086"
server_key_iv: "91f981ea65e5236443d13ef08cdbc5b3"
server_key_version: "encryptionkey@0"
I20260929 02:12:24.018805 27943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:24.030356 27943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:24.034435 27943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:24.034699 27943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:24.035561 27943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:24.036813 27943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:24.036926 27943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:24.037061 27943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:24.037137 27943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:24.070235 27943 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:38561
I20260929 02:12:24.070248 28070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:38561 every 8 connection(s)
I20260929 02:12:24.071568 27943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:12:24.081125 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 27943
I20260929 02:12:24.081337 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:12:24.081951 25442 external_mini_cluster.cc:1428] Setting key 156c2eeb0f92b239492cc6353dbb7aac
Sep 29 02:12:24 dist-test-slave-9k0h krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647943, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:24.091621 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:24.073998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:46299 (local address 127.24.216.190:39961)
0929 02:12:24.074462 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:24.074466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:24.075319 (+ 853us) server_negotiation.cc:407] Connection header received
0929 02:12:24.077123 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:24.077129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:24.077222 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:24.077346 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:24.078893 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:24.079662 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:24.081188 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:24.081700 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:24.081841 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:24.085720 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:24.085725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:24.085728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:24.085757 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:24.088470 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:24.089199 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:24.089209 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:24.089214 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:24.089330 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:24.089772 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:24.089777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:24.089781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:24.090042 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:24.090230 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:24.090992 (+ 762us) server_negotiation.cc:299] Negotiation successful
0929 02:12:24.091225 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":272,"threads_started":1}
I20260929 02:12:24.092540 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:24.074476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39961 (local address 127.24.216.129:46299)
0929 02:12:24.075152 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:24.075191 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:24.076827 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:24.077587 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:24.077597 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:24.078119 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:24.078729 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:24.078746 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:24.079810 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:24.079820 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:24.081022 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:24.081040 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:24.081924 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:24.081934 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:24.082290 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:24.083045 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:24.083073 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:24.085513 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:24.088629 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:24.088638 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:24.088654 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:24.089073 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:24.089439 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:24.089446 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:24.089449 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:24.089647 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:24.090196 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:24.090205 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:24.090621 (+ 416us) client_negotiation.cc:769] Sending connection context
0929 02:12:24.090888 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 02:12:24.091378 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":239,"threads_started":1}
I20260929 02:12:24.094255 28071 heartbeater.cc:347] Connected to a master server at 127.24.216.190:39961
I20260929 02:12:24.094592 28071 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:24.095415 28071 heartbeater.cc:511] Master 127.24.216.190:39961 requested a full tablet report, sending...
I20260929 02:12:24.097807 27882 ts_manager.cc:194] Registered new tserver with Master: 8a45ecdad58245bf847fdd0f39ebdfc7 (127.24.216.129:38561)
I20260929 02:12:24.100025 27882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:46299
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:12:24.125087 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39961
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:24.431947 28077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:24.432193 28077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:24.432353 28077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:24.439177 28077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:24.439287 28077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:24.439458 28077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:12:24.448418 28077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:24.449596 28077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:24.450910 28077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:24.463152 28082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:24.463179 28085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:24.463315 28083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:24.464114 28077 server_base.cc:1034] running on GCE node
I20260929 02:12:24.464663 28077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:24.465592 28077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:24.466822 28077 hybrid_clock.cc:648] HybridClock initialized: now 1790647944466785 us; error 57 us; skew 500 ppm
I20260929 02:12:24.467123 28077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:24 dist-test-slave-9k0h krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647944, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:24.469987 28077 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:12:24.471326 28077 webserver.cc:460] Webserver started at http://127.24.216.130:46029/ using document root <none> and password file <none>
I20260929 02:12:24.471961 28077 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:24.472092 28077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:24.472381 28077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:24.475158 28077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "4d7874e9bdd4481ea250968f6e3b0d5b"
format_stamp: "Formatted at 2026-09-29 02:12:24 on dist-test-slave-9k0h"
server_key: "7f207d249bfc19e28ffad2cf08670670"
server_key_iv: "46e0df29fab97fc6c63f9fd69a97add9"
server_key_version: "encryptionkey@0"
I20260929 02:12:24.475888 28077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "4d7874e9bdd4481ea250968f6e3b0d5b"
format_stamp: "Formatted at 2026-09-29 02:12:24 on dist-test-slave-9k0h"
server_key: "7f207d249bfc19e28ffad2cf08670670"
server_key_iv: "46e0df29fab97fc6c63f9fd69a97add9"
server_key_version: "encryptionkey@0"
I20260929 02:12:24.480962 28077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 02:12:24.484294 28092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:24.485553 28077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:12:24.485707 28077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "4d7874e9bdd4481ea250968f6e3b0d5b"
format_stamp: "Formatted at 2026-09-29 02:12:24 on dist-test-slave-9k0h"
server_key: "7f207d249bfc19e28ffad2cf08670670"
server_key_iv: "46e0df29fab97fc6c63f9fd69a97add9"
server_key_version: "encryptionkey@0"
I20260929 02:12:24.485898 28077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:24.497605 28077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:24.501236 28077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:24.501509 28077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:24.502485 28077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:24.503731 28077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:24.503839 28077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:24.503986 28077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:24.504055 28077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:24.534929 28077 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:44711
I20260929 02:12:24.534969 28204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:44711 every 8 connection(s)
I20260929 02:12:24.536166 28077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:12:24.541427 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 28077
I20260929 02:12:24.541620 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:12:24.542043 25442 external_mini_cluster.cc:1428] Setting key 550a570eb1d633c8a5d0f8e5224d2c5a
Sep 29 02:12:24 dist-test-slave-9k0h krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647944, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:24.556255 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:24.538487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:57291 (local address 127.24.216.190:39961)
0929 02:12:24.538643 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:24.538647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:24.539765 (+ 1118us) server_negotiation.cc:407] Connection header received
0929 02:12:24.541502 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:24.541507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:24.541593 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:24.541695 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:24.543297 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:24.544215 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:24.545663 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:24.546032 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:24.546155 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:24.549629 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:24.549634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:24.549640 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:24.549681 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:24.553192 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:24.553830 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:24.553838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:24.553844 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:24.554002 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:24.554429 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:24.554436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:24.554441 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:24.554796 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:24.554945 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:24.555537 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 02:12:24.555805 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:12:24.556828 28207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:24.538948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39961 (local address 127.24.216.130:57291)
0929 02:12:24.539620 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:24.539657 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:24.541244 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:24.541930 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:24.541938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:24.542486 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:24.543142 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:24.543159 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:24.544368 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:24.544372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:24.545529 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:24.545539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:24.546253 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:24.546262 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:24.546492 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:24.547152 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:24.547173 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:24.549419 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:24.553324 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:24.553330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:24.553341 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:24.553711 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:24.554107 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:24.554110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:24.554112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:24.554333 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:24.554908 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:24.554913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:24.555188 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 02:12:24.555399 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 02:12:24.555758 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":263,"threads_started":1}
I20260929 02:12:24.558260 28205 heartbeater.cc:347] Connected to a master server at 127.24.216.190:39961
I20260929 02:12:24.558591 28205 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:24.559520 28205 heartbeater.cc:511] Master 127.24.216.190:39961 requested a full tablet report, sending...
I20260929 02:12:24.561082 27882 ts_manager.cc:194] Registered new tserver with Master: 4d7874e9bdd4481ea250968f6e3b0d5b (127.24.216.130:44711)
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
I20260929 02:12:24.561883 27882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:57291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:12:24.583355 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39961
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:24.890450 28210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:24.890702 28210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:24.890789 28210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:24.897840 28210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:24.897933 28210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:24.898048 28210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:12:24.907804 28210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:24.908946 28210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:24.910130 28210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:24.922703 28218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:24.922744 28216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:24.922771 28215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:24.924124 28210 server_base.cc:1034] running on GCE node
I20260929 02:12:24.924608 28210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:24.925393 28210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:24.926575 28210 hybrid_clock.cc:648] HybridClock initialized: now 1790647944926523 us; error 46 us; skew 500 ppm
I20260929 02:12:24.926872 28210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:24 dist-test-slave-9k0h krb5kdc[27837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647944, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:24.929925 28210 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:12:24.931372 28210 webserver.cc:460] Webserver started at http://127.24.216.131:41487/ using document root <none> and password file <none>
I20260929 02:12:24.932041 28210 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:24.932163 28210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:24.932459 28210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:24.935261 28210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "b02abf722c0447ddbad79538e0ed02d3"
format_stamp: "Formatted at 2026-09-29 02:12:24 on dist-test-slave-9k0h"
server_key: "a5b7b7728d052aeb5ff2a2502e9df415"
server_key_iv: "68a50f8b5b5b4c123119d6fcb903e684"
server_key_version: "encryptionkey@0"
I20260929 02:12:24.936014 28210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "b02abf722c0447ddbad79538e0ed02d3"
format_stamp: "Formatted at 2026-09-29 02:12:24 on dist-test-slave-9k0h"
server_key: "a5b7b7728d052aeb5ff2a2502e9df415"
server_key_iv: "68a50f8b5b5b4c123119d6fcb903e684"
server_key_version: "encryptionkey@0"
I20260929 02:12:24.940879 28210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:12:24.944020 28225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:24.945173 28210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:24.945335 28210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "b02abf722c0447ddbad79538e0ed02d3"
format_stamp: "Formatted at 2026-09-29 02:12:24 on dist-test-slave-9k0h"
server_key: "a5b7b7728d052aeb5ff2a2502e9df415"
server_key_iv: "68a50f8b5b5b4c123119d6fcb903e684"
server_key_version: "encryptionkey@0"
I20260929 02:12:24.945514 28210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:24.962584 28210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:24.965992 28210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:24.966311 28210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:24.967149 28210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:24.968367 28210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:24.968461 28210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:24.968592 28210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:24.968655 28210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:24.999995 28210 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:33629
I20260929 02:12:25.000046 28337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:33629 every 8 connection(s)
I20260929 02:12:25.001209 28210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:12:25.008967 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 28210
I20260929 02:12:25.009159 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:12:25.009563 25442 external_mini_cluster.cc:1428] Setting key 8f9d9d58a72f00c175d8887a04b7de3f
Sep 29 02:12:25 dist-test-slave-9k0h krb5kdc[27837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647944, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:25.019893 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:25.003637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:47057 (local address 127.24.216.190:39961)
0929 02:12:25.003801 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:25.003805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:25.005023 (+ 1218us) server_negotiation.cc:407] Connection header received
0929 02:12:25.006866 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:25.006872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:25.006982 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:25.007096 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:25.008509 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:25.009338 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:25.011176 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:25.011751 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:25.011886 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:25.014982 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:25.014987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:25.014990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:25.015017 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:25.017398 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:25.018030 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:25.018033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:25.018036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:25.018193 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:25.018555 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:25.018558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:25.018560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:25.018814 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:25.018915 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:25.019500 (+ 585us) server_negotiation.cc:299] Negotiation successful
0929 02:12:25.019650 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:12:25.020733 28340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:25.004162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39961 (local address 127.24.216.131:47057)
0929 02:12:25.004873 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:25.004913 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:25.006600 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:25.007327 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:25.007335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:25.007813 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:25.008348 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:25.008364 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:25.009470 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:25.009475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:25.011039 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:25.011050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:25.011963 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:25.011972 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:25.012199 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:25.012819 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:25.012843 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:25.014826 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:25.017509 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:25.017515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:25.017525 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:25.017924 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:25.018289 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:25.018292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:25.018294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:25.018447 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:25.018914 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:25.018919 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:25.019204 (+ 285us) client_negotiation.cc:769] Sending connection context
0929 02:12:25.019428 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 02:12:25.019747 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":256,"threads_started":1}
I20260929 02:12:25.021929 28338 heartbeater.cc:347] Connected to a master server at 127.24.216.190:39961
I20260929 02:12:25.022325 28338 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:25.023128 28338 heartbeater.cc:511] Master 127.24.216.190:39961 requested a full tablet report, sending...
I20260929 02:12:25.024482 27882 ts_manager.cc:194] Registered new tserver with Master: b02abf722c0447ddbad79538e0ed02d3 (127.24.216.131:33629)
I20260929 02:12:25.025184 27882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:47057
I20260929 02:12:25.036187 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 02:12:25.065784 28073 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:12:25.063511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.24.216.190:39961)
0929 02:12:25.063658 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:25.063662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:25.064196 (+ 534us) server_negotiation.cc:407] Connection header received
0929 02:12:25.064326 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:25.064330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:25.064417 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:25.064500 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:25.065602 (+ 1102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43024: BlockingRecv error: recv got EOF from 127.0.0.1:43024 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:12:25.069144 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 27943
W20260929 02:12:25.071774 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 27943: Not authorized: Client connection negotiation failed: client connection to 127.24.216.129:38561: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:12:25.071943 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 27943
I20260929 02:12:25.086710 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 28077
W20260929 02:12:25.088939 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 28077: Not authorized: Client connection negotiation failed: client connection to 127.24.216.130:44711: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:12:25.089085 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 28077
I20260929 02:12:25.104001 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 28210
W20260929 02:12:25.106189 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 28210: Not authorized: Client connection negotiation failed: client connection to 127.24.216.131:33629: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:12:25.106305 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 28210
I20260929 02:12:25.120965 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 27850
I20260929 02:12:25.121994 27912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:25.224928 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 27850
2026-09-29T02:12:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3422 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:12:25 dist-test-slave-9k0h krb5kdc[28355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:12:25 dist-test-slave-9k0h krb5kdc[28355](info): set up 2 sockets
Sep 29 02:12:25 dist-test-slave-9k0h krb5kdc[28355](info): commencing operation
krb5kdc: starting...
W20260929 02:12:26.311398 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 02:12:26.490095 25442 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.178s user 0.000s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 02:12:26.689798 25442 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.199s user 0.003s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 02:12:26.948894 25442 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.259s user 0.000s sys 0.005s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/krb5kdc/test-user.keytab.
W20260929 02:12:27.209129 25442 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.260s user 0.000s sys 0.005s
Sep 29 02:12:27 dist-test-slave-9k0h krb5kdc[28355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:12:27.298290 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:33587
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:33587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:27.610334 28368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:27.610620 28368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:27.610787 28368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:27.617841 28368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:27.617935 28368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:27.618036 28368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:27.618129 28368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:27.618216 28368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:12:27.627547 28368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:33587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:33587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:27.628791 28368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:27.630110 28368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:27.647154 28374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:27.647207 28373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:27.647208 28376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:27.648095 28368 server_base.cc:1034] running on GCE node
I20260929 02:12:27.648914 28368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:27.650279 28368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:27.651494 28368 hybrid_clock.cc:648] HybridClock initialized: now 1790647947651456 us; error 50 us; skew 500 ppm
I20260929 02:12:27.651835 28368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:27 dist-test-slave-9k0h krb5kdc[28355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647947, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:27.660714 28368 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:12:27.662242 28368 webserver.cc:460] Webserver started at http://127.24.216.190:37819/ using document root <none> and password file <none>
I20260929 02:12:27.662896 28368 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:27.662992 28368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:27.663235 28368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:27.666009 28368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "688c99f616504ce590587c053c07089a"
format_stamp: "Formatted at 2026-09-29 02:12:27 on dist-test-slave-9k0h"
server_key: "d6c0745c8076c7dca5e34f8c31352108"
server_key_iv: "ff24e878188470c302b781d74f2ca311"
server_key_version: "encryptionkey@0"
I20260929 02:12:27.666795 28368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "688c99f616504ce590587c053c07089a"
format_stamp: "Formatted at 2026-09-29 02:12:27 on dist-test-slave-9k0h"
server_key: "d6c0745c8076c7dca5e34f8c31352108"
server_key_iv: "ff24e878188470c302b781d74f2ca311"
server_key_version: "encryptionkey@0"
I20260929 02:12:27.671617 28368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 02:12:27.674676 28383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:27.675864 28368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 02:12:27.676036 28368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "688c99f616504ce590587c053c07089a"
format_stamp: "Formatted at 2026-09-29 02:12:27 on dist-test-slave-9k0h"
server_key: "d6c0745c8076c7dca5e34f8c31352108"
server_key_iv: "ff24e878188470c302b781d74f2ca311"
server_key_version: "encryptionkey@0"
I20260929 02:12:27.676209 28368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:27.687043 28368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:27.690618 28368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:27.690876 28368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:27.708561 28368 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:33587
I20260929 02:12:27.708595 28434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:33587 every 8 connection(s)
I20260929 02:12:27.709857 28368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:27.713424 28435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:27.715188 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 28368
I20260929 02:12:27.715405 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:27.716181 25442 external_mini_cluster.cc:1428] Setting key fcea5e76aa5cedf68fc965a61b1f0b22
I20260929 02:12:27.727054 28435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a: Bootstrap starting.
Sep 29 02:12:27 dist-test-slave-9k0h krb5kdc[28355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:27.730803 28435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:27.731842 28435 log.cc:824] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:27.734505 28435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a: No bootstrap required, opened a new log
I20260929 02:12:27.737988 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:27.717965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.24.216.190:33587)
0929 02:12:27.718777 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:27.718791 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:27.718829 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:12:27.720338 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:27.720361 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:27.720739 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:27.721036 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:27.722874 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:27.724126 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:27.725884 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:27.726581 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:27.726704 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:27.731170 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:27.731180 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:27.731193 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:27.731228 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:27.734092 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:27.734805 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:27.734811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:27.734816 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:27.734995 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:27.735374 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:27.735380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:27.735383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:27.735886 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:27.736218 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:27.736639 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 02:12:27.737032 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":430,"threads_started":1}
I20260929 02:12:27.741525 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:12:27.742357 28435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688c99f616504ce590587c053c07089a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 33587 } }
I20260929 02:12:27.742681 28435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:27.742781 28435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688c99f616504ce590587c053c07089a, State: Initialized, Role: FOLLOWER
I20260929 02:12:27.743273 28435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688c99f616504ce590587c053c07089a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 33587 } }
I20260929 02:12:27.743499 28435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:27.743629 28435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:27.743798 28435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:27.745826 28435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688c99f616504ce590587c053c07089a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 33587 } }
I20260929 02:12:27.746342 28435 leader_election.cc:302] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 688c99f616504ce590587c053c07089a; no voters:
I20260929 02:12:27.746954 28435 leader_election.cc:288] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:27.747038 28440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:27.749390 28435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:27.750434 28440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [term 1 LEADER]: Becoming Leader. State: Replica: 688c99f616504ce590587c053c07089a, State: Running, Role: LEADER
I20260929 02:12:27.751080 28440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688c99f616504ce590587c053c07089a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 33587 } }
I20260929 02:12:27.754230 28442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 688c99f616504ce590587c053c07089a. Latest consensus state: current_term: 1 leader_uuid: "688c99f616504ce590587c053c07089a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688c99f616504ce590587c053c07089a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 33587 } } }
I20260929 02:12:27.754601 28442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:27.754330 28441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "688c99f616504ce590587c053c07089a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688c99f616504ce590587c053c07089a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 33587 } } }
I20260929 02:12:27.754709 28441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:27.755530 28449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:27.767843 28449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:27.792747 28449 catalog_manager.cc:1347] Generated new cluster ID: 8ce82f9b7801410fbdb0fd4d4aae73a5
I20260929 02:12:27.792900 28449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:27.808238 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:27.801066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.24.216.190:33587)
0929 02:12:27.801260 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:27.801269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:27.801851 (+ 582us) server_negotiation.cc:407] Connection header received
0929 02:12:27.802049 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:27.802054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:27.802118 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:27.802244 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:27.803159 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:27.803894 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:27.805131 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:27.805559 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:27.805696 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:27.806762 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:27.806768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:27.806772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:27.807452 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:27.807509 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:27.807537 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:27.807650 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:27.807794 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 02:12:27.807883 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:12:27.810498 28449 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:27.812079 28449 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:27.823616 28449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 688c99f616504ce590587c053c07089a: Generated new TSK 0
I20260929 02:12:27.824426 28449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:12:27.834365 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 28368
I20260929 02:12:27.835465 28430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:27.922466 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 28368
2026-09-29T02:12:27Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2691 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:12:28 dist-test-slave-9k0h krb5kdc[28469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:12:28 dist-test-slave-9k0h krb5kdc[28469](info): set up 2 sockets
Sep 29 02:12:28 dist-test-slave-9k0h krb5kdc[28469](info): commencing operation
krb5kdc: starting...
W20260929 02:12:28.978696 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:12:29 dist-test-slave-9k0h krb5kdc[28469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:12:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:12:29.171176 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:41981
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:41981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:29.484220 28482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:29.484575 28482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:29.484740 28482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:29.491717 28482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:29.491861 28482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:29.491987 28482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:29.492086 28482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:29.492164 28482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:12:29.501693 28482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:41981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:41981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:29.503123 28482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:29.504393 28482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:29.515847 28490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:29.515937 28487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:29.515940 28488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:29.516980 28482 server_base.cc:1034] running on GCE node
I20260929 02:12:29.517701 28482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:29.519050 28482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:29.520272 28482 hybrid_clock.cc:648] HybridClock initialized: now 1790647949520223 us; error 55 us; skew 500 ppm
I20260929 02:12:29.520615 28482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:29 dist-test-slave-9k0h krb5kdc[28469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647949, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:29.523538 28482 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:12:29.524938 28482 webserver.cc:460] Webserver started at http://127.24.216.190:40405/ using document root <none> and password file <none>
I20260929 02:12:29.525581 28482 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:29.525717 28482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:29.526031 28482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:29.528949 28482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "97539c9901974287826bb89a6f26f4f6"
format_stamp: "Formatted at 2026-09-29 02:12:29 on dist-test-slave-9k0h"
server_key: "001c6e6e94099100fb3d23de177acce1"
server_key_iv: "0d66b3c95b5d938ced496885038a49e9"
server_key_version: "encryptionkey@0"
I20260929 02:12:29.529670 28482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "97539c9901974287826bb89a6f26f4f6"
format_stamp: "Formatted at 2026-09-29 02:12:29 on dist-test-slave-9k0h"
server_key: "001c6e6e94099100fb3d23de177acce1"
server_key_iv: "0d66b3c95b5d938ced496885038a49e9"
server_key_version: "encryptionkey@0"
I20260929 02:12:29.534677 28482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:12:29.537865 28497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:29.539131 28482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:29.539315 28482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "97539c9901974287826bb89a6f26f4f6"
format_stamp: "Formatted at 2026-09-29 02:12:29 on dist-test-slave-9k0h"
server_key: "001c6e6e94099100fb3d23de177acce1"
server_key_iv: "0d66b3c95b5d938ced496885038a49e9"
server_key_version: "encryptionkey@0"
I20260929 02:12:29.539507 28482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:29.561144 28482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:29.564699 28482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:29.564968 28482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:29.582496 28482 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:41981
I20260929 02:12:29.582535 28548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:41981 every 8 connection(s)
I20260929 02:12:29.583804 28482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:29.587352 28549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:29.588861 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 28482
I20260929 02:12:29.589087 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:29.589931 25442 external_mini_cluster.cc:1428] Setting key 2a364444be23bb2ad11709f43d50e6cb
I20260929 02:12:29.600028 28549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6: Bootstrap starting.
Sep 29 02:12:29 dist-test-slave-9k0h krb5kdc[28469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:29.603192 28549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:29.604389 28549 log.cc:824] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:29.607362 28549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6: No bootstrap required, opened a new log
I20260929 02:12:29.611238 28552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:29.592084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.24.216.190:41981)
0929 02:12:29.592841 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:29.592852 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:29.592893 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:12:29.594425 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:29.594448 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:29.594822 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:29.595166 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:29.596646 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:29.597936 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:29.599181 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:29.599830 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:29.600026 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:29.604144 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:29.604152 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:29.604166 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:29.604197 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:29.607381 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:29.608163 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:29.608169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:29.608174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:29.608337 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:29.608738 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:29.608741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:29.608743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:29.609221 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:29.609450 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:29.609887 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 02:12:29.610275 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":339,"threads_started":1}
I20260929 02:12:29.615422 28549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97539c9901974287826bb89a6f26f4f6" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41981 } }
I20260929 02:12:29.615768 28549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:29.615883 28549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97539c9901974287826bb89a6f26f4f6, State: Initialized, Role: FOLLOWER
I20260929 02:12:29.616451 28549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97539c9901974287826bb89a6f26f4f6" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41981 } }
I20260929 02:12:29.616664 28549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:29.616812 28549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:29.616971 28549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:29.619028 28549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97539c9901974287826bb89a6f26f4f6" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41981 } }
I20260929 02:12:29.619541 28549 leader_election.cc:302] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97539c9901974287826bb89a6f26f4f6; no voters:
I20260929 02:12:29.620172 28549 leader_election.cc:288] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:29.620298 28554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:29.620965 28554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [term 1 LEADER]: Becoming Leader. State: Replica: 97539c9901974287826bb89a6f26f4f6, State: Running, Role: LEADER
I20260929 02:12:29.621456 28554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97539c9901974287826bb89a6f26f4f6" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41981 } }
I20260929 02:12:29.622121 28549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:29.623886 28555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97539c9901974287826bb89a6f26f4f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97539c9901974287826bb89a6f26f4f6" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41981 } } }
I20260929 02:12:29.624163 28555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:29.624992 28560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:29.624846 28556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97539c9901974287826bb89a6f26f4f6. Latest consensus state: current_term: 1 leader_uuid: "97539c9901974287826bb89a6f26f4f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97539c9901974287826bb89a6f26f4f6" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41981 } } }
I20260929 02:12:29.625927 28556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:29.630829 28560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:29.639557 28560 catalog_manager.cc:1347] Generated new cluster ID: 611c5174bf844d6c86b7382cfa597582
I20260929 02:12:29.639664 28560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:29.663384 28560 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:29.665174 28560 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:29.677095 28560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97539c9901974287826bb89a6f26f4f6: Generated new TSK 0
I20260929 02:12:29.678113 28560 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:12:29.750648 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41981
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:30.193101 28575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:30.193391 28575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:30.193478 28575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:30.200536 28575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:30.200627 28575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:30.200733 28575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:12:30.209661 28575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:30.210827 28575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:30.212003 28575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:30.224808 28580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:30.224849 28581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:30.224895 28583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:30.227415 28575 server_base.cc:1034] running on GCE node
I20260929 02:12:30.227908 28575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:30.228766 28575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:30.229950 28575 hybrid_clock.cc:648] HybridClock initialized: now 1790647950229907 us; error 34 us; skew 500 ppm
I20260929 02:12:30.230288 28575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:30 dist-test-slave-9k0h krb5kdc[28469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647950, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:30.233229 28575 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:12:30.234647 28575 webserver.cc:460] Webserver started at http://127.24.216.129:35451/ using document root <none> and password file <none>
I20260929 02:12:30.235347 28575 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:30.235476 28575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:30.235771 28575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:30.238536 28575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "765a921cc50c4ff88998ec2be48ea95f"
format_stamp: "Formatted at 2026-09-29 02:12:30 on dist-test-slave-9k0h"
server_key: "112b0e4cc47cad6047747ffc31db6fe4"
server_key_iv: "3014187d68d3cff9629e694f2fb211d2"
server_key_version: "encryptionkey@0"
I20260929 02:12:30.239252 28575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "765a921cc50c4ff88998ec2be48ea95f"
format_stamp: "Formatted at 2026-09-29 02:12:30 on dist-test-slave-9k0h"
server_key: "112b0e4cc47cad6047747ffc31db6fe4"
server_key_iv: "3014187d68d3cff9629e694f2fb211d2"
server_key_version: "encryptionkey@0"
I20260929 02:12:30.244009 28575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:12:30.246989 28590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:30.248222 28575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 02:12:30.248385 28575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "765a921cc50c4ff88998ec2be48ea95f"
format_stamp: "Formatted at 2026-09-29 02:12:30 on dist-test-slave-9k0h"
server_key: "112b0e4cc47cad6047747ffc31db6fe4"
server_key_iv: "3014187d68d3cff9629e694f2fb211d2"
server_key_version: "encryptionkey@0"
I20260929 02:12:30.248610 28575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:30.271987 28575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:30.275972 28575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:30.276225 28575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:30.277146 28575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:30.278539 28575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:30.278614 28575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:30.278698 28575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:30.278741 28575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:30.311839 28575 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:44693
I20260929 02:12:30.311877 28702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:44693 every 8 connection(s)
I20260929 02:12:30.313100 28575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:12:30.314795 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 28575
I20260929 02:12:30.314986 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:12:30.315822 25442 external_mini_cluster.cc:1428] Setting key 3b012466ee56874a6d5e55d61bf145ce
Sep 29 02:12:30 dist-test-slave-9k0h krb5kdc[28469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647950, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:30.335078 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:30.315816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:35907 (local address 127.24.216.190:41981)
0929 02:12:30.316262 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:30.316266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:30.317552 (+ 1286us) server_negotiation.cc:407] Connection header received
0929 02:12:30.319600 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:30.319608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:30.319694 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:30.319816 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:30.321311 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:30.322518 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:30.324836 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:30.325217 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:30.325322 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:30.329571 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:30.329576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:30.329579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:30.329606 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:30.332155 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:30.332849 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:30.332857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:30.332861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:30.332974 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:30.333378 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:30.333385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:30.333389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:30.333630 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:30.333773 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:30.334508 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 02:12:30.334715 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":245,"threads_started":1}
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
I20260929 02:12:30.336076 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:30.316459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:41981 (local address 127.24.216.129:35907)
0929 02:12:30.317364 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:30.317422 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:30.319278 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:30.320016 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:30.320024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:30.320544 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:30.321134 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:30.321152 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:30.322683 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:30.322687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:30.324688 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:30.324705 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:30.325448 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:30.325459 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:30.325817 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:30.326592 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:30.326629 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:30.329366 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:30.332309 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:30.332317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:30.332332 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:30.332730 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:30.333076 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:30.333083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:30.333086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:30.333268 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:30.333767 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:30.333774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:30.334094 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 02:12:30.334420 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 02:12:30.334927 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":357,"threads_started":1}
I20260929 02:12:30.337728 28703 heartbeater.cc:347] Connected to a master server at 127.24.216.190:41981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
I20260929 02:12:30.338029 28703 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:30.338923 28703 heartbeater.cc:511] Master 127.24.216.190:41981 requested a full tablet report, sending...
I20260929 02:12:30.341228 28514 ts_manager.cc:194] Registered new tserver with Master: 765a921cc50c4ff88998ec2be48ea95f (127.24.216.129:44693)
I20260929 02:12:30.342839 28514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:35907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:12:30.362412 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41981
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:30.673904 28709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:30.674247 28709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:30.674401 28709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:30.681339 28709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:30.681463 28709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:30.681655 28709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:12:30.691351 28709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:30.692580 28709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:30.693861 28709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:30.706647 28714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:30.706646 28717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:30.706583 28715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:30.707297 28709 server_base.cc:1034] running on GCE node
I20260929 02:12:30.707813 28709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:30.708655 28709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:30.709870 28709 hybrid_clock.cc:648] HybridClock initialized: now 1790647950709821 us; error 58 us; skew 500 ppm
I20260929 02:12:30.710207 28709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:30 dist-test-slave-9k0h krb5kdc[28469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647950, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:30.713133 28709 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:12:30.714556 28709 webserver.cc:460] Webserver started at http://127.24.216.130:45335/ using document root <none> and password file <none>
I20260929 02:12:30.715246 28709 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:30.715353 28709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:30.715615 28709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:30.718328 28709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "00d24ea2e3f8462497a40a267ddadc61"
format_stamp: "Formatted at 2026-09-29 02:12:30 on dist-test-slave-9k0h"
server_key: "bbe3116b9500bbbec840815100a2d9ca"
server_key_iv: "4bdf0e2a32777975f902c25531000289"
server_key_version: "encryptionkey@0"
I20260929 02:12:30.719014 28709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "00d24ea2e3f8462497a40a267ddadc61"
format_stamp: "Formatted at 2026-09-29 02:12:30 on dist-test-slave-9k0h"
server_key: "bbe3116b9500bbbec840815100a2d9ca"
server_key_iv: "4bdf0e2a32777975f902c25531000289"
server_key_version: "encryptionkey@0"
I20260929 02:12:30.723807 28709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 02:12:30.726972 28724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:30.728259 28709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:12:30.728426 28709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "00d24ea2e3f8462497a40a267ddadc61"
format_stamp: "Formatted at 2026-09-29 02:12:30 on dist-test-slave-9k0h"
server_key: "bbe3116b9500bbbec840815100a2d9ca"
server_key_iv: "4bdf0e2a32777975f902c25531000289"
server_key_version: "encryptionkey@0"
I20260929 02:12:30.728615 28709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:30.750437 28709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:30.754184 28709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:30.754464 28709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:30.755360 28709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:30.756565 28709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:30.756655 28709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:30.756810 28709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:30.756876 28709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:30.787398 28709 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:37625
I20260929 02:12:30.787477 28836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:37625 every 8 connection(s)
I20260929 02:12:30.788693 28709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:12:30.789265 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 28709
I20260929 02:12:30.789458 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:12:30.790272 25442 external_mini_cluster.cc:1428] Setting key 91c93b41bf2a9194e26aab7b2a88f3e0
Sep 29 02:12:30 dist-test-slave-9k0h krb5kdc[28469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647950, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
I20260929 02:12:30.808531 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:30.791254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:41039 (local address 127.24.216.190:41981)
0929 02:12:30.791448 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:30.791454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:30.792745 (+ 1291us) server_negotiation.cc:407] Connection header received
0929 02:12:30.794527 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:30.794536 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:30.794630 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:30.794739 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:30.796077 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:30.797202 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:30.799393 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:30.799806 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:30.799933 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:30.803152 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:30.803157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:30.803160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:30.803189 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:30.805702 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:30.806440 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:30.806445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:30.806453 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:30.806564 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:30.806950 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:30.806954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:30.806958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:30.807190 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:30.807372 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:30.807983 (+ 611us) server_negotiation.cc:299] Negotiation successful
0929 02:12:30.808192 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:12:30.809386 28839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:30.791785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:41981 (local address 127.24.216.130:41039)
0929 02:12:30.792580 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:30.792628 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:30.794256 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:30.794909 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:30.794917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:30.795380 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:30.795918 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:30.795933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:30.797357 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:30.797361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:30.799237 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:30.799248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:30.800028 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:30.800035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:30.800243 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:30.800935 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:30.800956 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:30.802955 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:30.805860 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:30.805868 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:30.805881 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:30.806328 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:30.806666 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:30.806672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:30.806675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:30.806841 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:30.807337 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:30.807346 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:30.807651 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 02:12:30.807887 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 02:12:30.808369 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":320,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
I20260929 02:12:30.810750 28837 heartbeater.cc:347] Connected to a master server at 127.24.216.190:41981
I20260929 02:12:30.811094 28837 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:30.811784 28837 heartbeater.cc:511] Master 127.24.216.190:41981 requested a full tablet report, sending...
I20260929 02:12:30.813117 28514 ts_manager.cc:194] Registered new tserver with Master: 00d24ea2e3f8462497a40a267ddadc61 (127.24.216.130:37625)
I20260929 02:12:30.813894 28514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:41039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:12:30.834417 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41981
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:31.145915 28842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:31.146262 28842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:31.146384 28842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:31.153688 28842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:31.153791 28842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:31.153913 28842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:12:31.163642 28842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:31.164958 28842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:31.166276 28842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:31.179672 28847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:31.179699 28848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:31.179967 28850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:31.180411 28842 server_base.cc:1034] running on GCE node
I20260929 02:12:31.181033 28842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:31.181946 28842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:31.183156 28842 hybrid_clock.cc:648] HybridClock initialized: now 1790647951183101 us; error 55 us; skew 500 ppm
I20260929 02:12:31.183485 28842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:31 dist-test-slave-9k0h krb5kdc[28469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647951, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:31.186508 28842 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:12:31.187762 28842 webserver.cc:460] Webserver started at http://127.24.216.131:44511/ using document root <none> and password file <none>
I20260929 02:12:31.188365 28842 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:31.188462 28842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:31.188716 28842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:31.191519 28842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "2451a07b6ccc4f7ebb437873e08c3286"
format_stamp: "Formatted at 2026-09-29 02:12:31 on dist-test-slave-9k0h"
server_key: "9c2d9e1c14cfeb63e317bbce68c22d71"
server_key_iv: "5d50c4b5599c08baa9332b249beb8b77"
server_key_version: "encryptionkey@0"
I20260929 02:12:31.192204 28842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "2451a07b6ccc4f7ebb437873e08c3286"
format_stamp: "Formatted at 2026-09-29 02:12:31 on dist-test-slave-9k0h"
server_key: "9c2d9e1c14cfeb63e317bbce68c22d71"
server_key_iv: "5d50c4b5599c08baa9332b249beb8b77"
server_key_version: "encryptionkey@0"
I20260929 02:12:31.197440 28842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:12:31.202728 28857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:31.204528 28842 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:12:31.204696 28842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "2451a07b6ccc4f7ebb437873e08c3286"
format_stamp: "Formatted at 2026-09-29 02:12:31 on dist-test-slave-9k0h"
server_key: "9c2d9e1c14cfeb63e317bbce68c22d71"
server_key_iv: "5d50c4b5599c08baa9332b249beb8b77"
server_key_version: "encryptionkey@0"
I20260929 02:12:31.204968 28842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:31.232740 28842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:31.237141 28842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:31.237461 28842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:31.238776 28842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:31.240558 28842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:31.240648 28842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:31.240772 28842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:31.240859 28842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:31.272360 28842 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:32977
I20260929 02:12:31.272406 28969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:32977 every 8 connection(s)
I20260929 02:12:31.273649 28842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:12:31.281591 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 28842
I20260929 02:12:31.281754 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:12:31.282346 25442 external_mini_cluster.cc:1428] Setting key b607b4363ee5c149c93d91e442e8075b
Sep 29 02:12:31 dist-test-slave-9k0h krb5kdc[28469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647951, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:31.292694 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:31.276168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:57511 (local address 127.24.216.190:41981)
0929 02:12:31.276343 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:31.276347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:31.277509 (+ 1162us) server_negotiation.cc:407] Connection header received
0929 02:12:31.279294 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:31.279298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:31.279383 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:31.279465 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:31.280856 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:31.281938 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:31.283797 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:31.284367 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:31.284460 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:31.287842 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:31.287847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:31.287850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:31.287879 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:31.290238 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:31.290850 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:31.290853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:31.290855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:31.290961 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:31.291316 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:31.291323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:31.291326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:31.291601 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:31.291710 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:31.292308 (+ 598us) server_negotiation.cc:299] Negotiation successful
0929 02:12:31.292454 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:12:31.293598 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:31.276656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:41981 (local address 127.24.216.131:57511)
0929 02:12:31.277352 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:31.277394 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:31.279033 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:31.279666 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:31.279675 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:31.280152 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:31.280706 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:31.280723 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:31.282061 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:31.282065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:31.283648 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:31.283659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:31.284579 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:31.284586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:31.284860 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:31.285586 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:31.285612 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:31.287674 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:31.290357 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:31.290363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:31.290375 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:31.290753 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:31.291041 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:31.291044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:31.291046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:31.291204 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:31.291700 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:31.291705 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:31.291981 (+ 276us) client_negotiation.cc:769] Sending connection context
0929 02:12:31.292205 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 02:12:31.292567 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":264,"threads_started":1}
I20260929 02:12:31.294934 28970 heartbeater.cc:347] Connected to a master server at 127.24.216.190:41981
I20260929 02:12:31.295223 28970 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:31.296119 28970 heartbeater.cc:511] Master 127.24.216.190:41981 requested a full tablet report, sending...
I20260929 02:12:31.297637 28514 ts_manager.cc:194] Registered new tserver with Master: 2451a07b6ccc4f7ebb437873e08c3286 (127.24.216.131:32977)
I20260929 02:12:31.298454 28514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:57511
I20260929 02:12:31.309185 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:12:31 dist-test-slave-9k0h krb5kdc[28469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647951, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260929 02:12:31.345770 28703 heartbeater.cc:503] Master 127.24.216.190:41981 was elected leader, sending a full tablet report...
Sep 29 02:12:31 dist-test-slave-9k0h krb5kdc[28469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647951, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:31.356956 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:31.343245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.24.216.190:41981)
0929 02:12:31.343417 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:31.343423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:31.343902 (+ 479us) server_negotiation.cc:407] Connection header received
0929 02:12:31.344066 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:31.344072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:31.344143 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:31.344236 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:31.345492 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:31.346585 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:31.347824 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:31.348249 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:31.348379 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:31.352022 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:31.352028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:31.352036 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:31.352080 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:31.354451 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:31.355040 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:31.355046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:31.355050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:31.355190 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:31.355546 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:31.355553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:31.355556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:31.355867 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:31.356108 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:31.356401 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 02:12:31.356633 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 02:12:31.357474 28514 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38830
I20260929 02:12:31.359136 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 28575
Sep 29 02:12:31 dist-test-slave-9k0h krb5kdc[28469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647951, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:12:31.373539 28981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:31.360095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.24.216.129:44693)
0929 02:12:31.360664 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:31.360672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:31.360695 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:12:31.360745 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:31.360751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:31.360961 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:31.361091 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:31.362096 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:31.363147 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:31.364258 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:31.364803 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:31.364897 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:31.368498 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:31.368509 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:31.368524 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:31.368561 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:31.371331 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:31.371904 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:31.371911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:31.371913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:31.372030 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:31.372316 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:31.372319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:31.372321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:31.372626 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:31.372843 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:31.373020 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 02:12:31.373202 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":297,"threads_started":1}
W20260929 02:12:31.374810 28698 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40118
W20260929 02:12:31.375850 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 28575: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:12:31.376012 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 28575
I20260929 02:12:31.392503 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 28709
Sep 29 02:12:31 dist-test-slave-9k0h krb5kdc[28469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647951, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:12:31.407404 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:31.393304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.24.216.130:37625)
0929 02:12:31.393873 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:31.393880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:31.393906 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:12:31.393965 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:31.393972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:31.394240 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:31.394371 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:31.395507 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:31.396511 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:31.397673 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:31.398310 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:31.398400 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:31.402273 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:31.402293 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:31.402319 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:31.402361 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:31.404891 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:31.405460 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:31.405466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:31.405468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:31.405582 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:31.405881 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:31.405884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:31.405886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:31.406236 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:31.406460 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:31.406834 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 02:12:31.407093 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":321,"threads_started":1}
W20260929 02:12:31.408614 28832 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55536
W20260929 02:12:31.409642 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 28709: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:12:31.409781 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 28709
I20260929 02:12:31.424954 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 28842
Sep 29 02:12:31 dist-test-slave-9k0h krb5kdc[28469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647951, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:12:31.440104 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:31.425831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40714 (local address 127.24.216.131:32977)
0929 02:12:31.426518 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:31.426524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:31.426550 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:12:31.426605 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:31.426611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:31.426836 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:31.426986 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:31.428350 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:31.429380 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:31.430714 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:31.431246 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:31.431377 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:31.435371 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:31.435381 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:31.435394 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:31.435427 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:31.437823 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:31.438472 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:31.438479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:31.438481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:31.438601 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:31.438893 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:31.438896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:31.438898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:31.439199 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:31.439440 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:31.439642 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 02:12:31.439828 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":381,"threads_started":1}
W20260929 02:12:31.441373 28965 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40714
W20260929 02:12:31.442508 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 28842: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:12:31.442667 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 28842
I20260929 02:12:31.457859 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 28482
I20260929 02:12:31.459010 28544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:31.560484 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 28482
2026-09-29T02:12:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3644 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:12:31 dist-test-slave-9k0h krb5kdc[28987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:12:31 dist-test-slave-9k0h krb5kdc[28987](info): set up 2 sockets
Sep 29 02:12:31 dist-test-slave-9k0h krb5kdc[28987](info): commencing operation
krb5kdc: starting...
W20260929 02:12:32.642813 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:12:32 dist-test-slave-9k0h krb5kdc[28987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:12:32.832957 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:39397
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:34975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:39397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:33.141322 29000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:33.141625 29000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:33.141741 29000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:33.148716 29000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:33.148826 29000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:33.148887 29000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:33.148983 29000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:33.149055 29000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:12:33.159099 29000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:34975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:39397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:39397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:33.160671 29000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:33.162010 29000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:33.173506 29005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:33.173461 29008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:33.173497 29006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:33.174424 29000 server_base.cc:1034] running on GCE node
I20260929 02:12:33.175117 29000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:33.176407 29000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:33.177615 29000 hybrid_clock.cc:648] HybridClock initialized: now 1790647953177574 us; error 53 us; skew 500 ppm
I20260929 02:12:33.177930 29000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:33 dist-test-slave-9k0h krb5kdc[28987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647953, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:33.186962 29000 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:12:33.188328 29000 webserver.cc:460] Webserver started at http://127.24.216.190:34137/ using document root <none> and password file <none>
I20260929 02:12:33.188949 29000 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:33.189038 29000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:33.189277 29000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:33.192005 29000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "627a3d46975e44c28dde306ecff21b92"
format_stamp: "Formatted at 2026-09-29 02:12:33 on dist-test-slave-9k0h"
server_key: "f1c6035451fe72a4691a16c98ad2cf28"
server_key_iv: "39648954941e8eb081c2d23ef397a7d4"
server_key_version: "encryptionkey@0"
I20260929 02:12:33.192662 29000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "627a3d46975e44c28dde306ecff21b92"
format_stamp: "Formatted at 2026-09-29 02:12:33 on dist-test-slave-9k0h"
server_key: "f1c6035451fe72a4691a16c98ad2cf28"
server_key_iv: "39648954941e8eb081c2d23ef397a7d4"
server_key_version: "encryptionkey@0"
I20260929 02:12:33.197427 29000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 02:12:33.200701 29015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:33.201984 29000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:12:33.202118 29000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "627a3d46975e44c28dde306ecff21b92"
format_stamp: "Formatted at 2026-09-29 02:12:33 on dist-test-slave-9k0h"
server_key: "f1c6035451fe72a4691a16c98ad2cf28"
server_key_iv: "39648954941e8eb081c2d23ef397a7d4"
server_key_version: "encryptionkey@0"
I20260929 02:12:33.202340 29000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:33.221400 29000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:33.225602 29000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:33.225867 29000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:33.241855 29000 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:39397
I20260929 02:12:33.241874 29066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:39397 every 8 connection(s)
I20260929 02:12:33.243155 29000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:33.246768 29067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:33.249897 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 29000
I20260929 02:12:33.250101 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:33.250757 25442 external_mini_cluster.cc:1428] Setting key dbec297e7bd4588e43303ce3a0f8e502
I20260929 02:12:33.259366 29067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92: Bootstrap starting.
I20260929 02:12:33.262871 29067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:33.264185 29067 log.cc:824] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:12:33 dist-test-slave-9k0h krb5kdc[28987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:33.267294 29067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92: No bootstrap required, opened a new log
I20260929 02:12:33.273957 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:33.252636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40218 (local address 127.24.216.190:39397)
0929 02:12:33.253430 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:33.253444 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:33.253493 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:12:33.255302 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:33.255330 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:33.255852 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:33.256281 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:33.257585 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:33.259545 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:33.260760 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:33.261731 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:33.261918 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:33.266777 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:33.266784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:33.266804 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:33.266836 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:33.269951 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:33.270767 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:33.270773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:33.270779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:33.270972 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:33.271357 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:33.271360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:33.271362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:33.271886 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:33.272143 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:33.272538 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 02:12:33.272926 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":343,"threads_started":1}
I20260929 02:12:33.275794 29067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627a3d46975e44c28dde306ecff21b92" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39397 } }
I20260929 02:12:33.276156 29067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:33.276266 29067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627a3d46975e44c28dde306ecff21b92, State: Initialized, Role: FOLLOWER
I20260929 02:12:33.276790 29067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627a3d46975e44c28dde306ecff21b92" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39397 } }
I20260929 02:12:33.277033 29067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:33.277165 29067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:33.277321 29067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:33.279487 29067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627a3d46975e44c28dde306ecff21b92" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39397 } }
I20260929 02:12:33.279953 29067 leader_election.cc:302] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 627a3d46975e44c28dde306ecff21b92; no voters:
I20260929 02:12:33.280555 29067 leader_election.cc:288] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:33.280666 29072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:33.281566 29072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [term 1 LEADER]: Becoming Leader. State: Replica: 627a3d46975e44c28dde306ecff21b92, State: Running, Role: LEADER
I20260929 02:12:33.282119 29072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627a3d46975e44c28dde306ecff21b92" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39397 } }
I20260929 02:12:33.282631 29067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:33.284993 29073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "627a3d46975e44c28dde306ecff21b92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627a3d46975e44c28dde306ecff21b92" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39397 } } }
I20260929 02:12:33.285490 29073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:33.285460 29074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 627a3d46975e44c28dde306ecff21b92. Latest consensus state: current_term: 1 leader_uuid: "627a3d46975e44c28dde306ecff21b92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627a3d46975e44c28dde306ecff21b92" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39397 } } }
I20260929 02:12:33.285758 29074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:33.291563 29079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:33.296275 29079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:33.303997 29079 catalog_manager.cc:1347] Generated new cluster ID: a678f8ebe27c4bdd849dd14da822b28d
I20260929 02:12:33.304111 29079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:33.310671 29079 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:33.311892 29079 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:33.324318 29079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 627a3d46975e44c28dde306ecff21b92: Generated new TSK 0
I20260929 02:12:33.325049 29079 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:12:33.369251 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39397
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:34975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:33.684126 29093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:33.684377 29093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:33.684552 29093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:33.691610 29093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:33.691754 29093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:33.691920 29093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:12:33.701082 29093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:34975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:33.702394 29093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:33.703714 29093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:33.716265 29098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:33.716297 29101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:33.716262 29099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:33.716949 29093 server_base.cc:1034] running on GCE node
I20260929 02:12:33.717480 29093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:33.718410 29093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:33.719595 29093 hybrid_clock.cc:648] HybridClock initialized: now 1790647953719553 us; error 35 us; skew 500 ppm
I20260929 02:12:33.719878 29093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:33 dist-test-slave-9k0h krb5kdc[28987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647953, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:33.722905 29093 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:12:33.724210 29093 webserver.cc:460] Webserver started at http://127.24.216.129:40541/ using document root <none> and password file <none>
I20260929 02:12:33.724843 29093 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:33.724960 29093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:33.725255 29093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:33.727967 29093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "73b1dfc976f04fe2b260d7dcf8b6e011"
format_stamp: "Formatted at 2026-09-29 02:12:33 on dist-test-slave-9k0h"
server_key: "7b4d1a3263120bd4895b0c174f6b9813"
server_key_iv: "2e6e29e8803d83e1d744390013657c9c"
server_key_version: "encryptionkey@0"
I20260929 02:12:33.728694 29093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "73b1dfc976f04fe2b260d7dcf8b6e011"
format_stamp: "Formatted at 2026-09-29 02:12:33 on dist-test-slave-9k0h"
server_key: "7b4d1a3263120bd4895b0c174f6b9813"
server_key_iv: "2e6e29e8803d83e1d744390013657c9c"
server_key_version: "encryptionkey@0"
I20260929 02:12:33.733453 29093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:12:33.736560 29108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:33.737735 29093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:33.737879 29093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "73b1dfc976f04fe2b260d7dcf8b6e011"
format_stamp: "Formatted at 2026-09-29 02:12:33 on dist-test-slave-9k0h"
server_key: "7b4d1a3263120bd4895b0c174f6b9813"
server_key_iv: "2e6e29e8803d83e1d744390013657c9c"
server_key_version: "encryptionkey@0"
I20260929 02:12:33.738067 29093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:33.752525 29093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:33.756470 29093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:33.756737 29093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:33.757633 29093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:33.758895 29093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:33.758963 29093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:33.759119 29093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:33.759177 29093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:33.791388 29093 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:41031
I20260929 02:12:33.791409 29220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:41031 every 8 connection(s)
I20260929 02:12:33.792695 29093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:12:33.796809 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 29093
I20260929 02:12:33.797032 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:12:33.797858 25442 external_mini_cluster.cc:1428] Setting key 51673018493821fea371263d6541b239
Sep 29 02:12:33 dist-test-slave-9k0h krb5kdc[28987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647953, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:33.813946 29223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:33.795071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:53883 (local address 127.24.216.190:39397)
0929 02:12:33.795477 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:33.795483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:33.796585 (+ 1102us) server_negotiation.cc:407] Connection header received
0929 02:12:33.798950 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:33.798957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:33.799016 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:33.799107 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:33.801046 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:33.801723 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:33.804186 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:33.804856 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:33.804976 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:33.808659 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:33.808677 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:33.808681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:33.808725 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:33.811078 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:33.811740 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:33.811745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:33.811752 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:33.811858 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:33.812337 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:33.812341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:33.812345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:33.812597 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:33.812782 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:33.813395 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 02:12:33.813592 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":195,"threads_started":1}
I20260929 02:12:33.814914 29224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:33.795674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39397 (local address 127.24.216.129:53883)
0929 02:12:33.796386 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:33.796447 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:33.798708 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:33.799386 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:33.799397 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:33.800019 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:33.800846 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:33.800873 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:33.801888 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:33.801896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:33.804048 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:33.804066 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:33.805075 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:33.805084 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:33.805437 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:33.806286 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:33.806335 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:33.808466 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:33.811221 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:33.811229 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:33.811244 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:33.811617 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:33.811956 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:33.811962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:33.811967 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:33.812209 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:33.812728 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:33.812735 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:33.813065 (+ 330us) client_negotiation.cc:769] Sending connection context
0929 02:12:33.813309 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 02:12:33.813800 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":239,"threads_started":1}
I20260929 02:12:33.816483 29221 heartbeater.cc:347] Connected to a master server at 127.24.216.190:39397
I20260929 02:12:33.816829 29221 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
I20260929 02:12:33.817692 29221 heartbeater.cc:511] Master 127.24.216.190:39397 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
I20260929 02:12:33.819805 29032 ts_manager.cc:194] Registered new tserver with Master: 73b1dfc976f04fe2b260d7dcf8b6e011 (127.24.216.129:41031)
I20260929 02:12:33.821208 29032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:53883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:12:33.841504 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39397
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:34975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:34.153908 29227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:34.154244 29227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:34.154407 29227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:34.162007 29227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:34.162150 29227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:34.162422 29227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:12:34.172595 29227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:34975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:34.173882 29227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:34.175220 29227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:34.187829 29235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:34.187875 29232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:34.188429 29233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:34.188680 29227 server_base.cc:1034] running on GCE node
I20260929 02:12:34.189266 29227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:34.190199 29227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:34.191431 29227 hybrid_clock.cc:648] HybridClock initialized: now 1790647954191385 us; error 55 us; skew 500 ppm
I20260929 02:12:34.191720 29227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:34 dist-test-slave-9k0h krb5kdc[28987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647954, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:34.194571 29227 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:12:34.195883 29227 webserver.cc:460] Webserver started at http://127.24.216.130:33143/ using document root <none> and password file <none>
I20260929 02:12:34.196512 29227 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:34.196652 29227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:34.196942 29227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:34.199757 29227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "108165158d11456b9c828ccb9c05bff0"
format_stamp: "Formatted at 2026-09-29 02:12:34 on dist-test-slave-9k0h"
server_key: "35c8e128a9ca9297a5544a27c703a28a"
server_key_iv: "263f6b576c44941ce1e555492be8c19c"
server_key_version: "encryptionkey@0"
I20260929 02:12:34.200505 29227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "108165158d11456b9c828ccb9c05bff0"
format_stamp: "Formatted at 2026-09-29 02:12:34 on dist-test-slave-9k0h"
server_key: "35c8e128a9ca9297a5544a27c703a28a"
server_key_iv: "263f6b576c44941ce1e555492be8c19c"
server_key_version: "encryptionkey@0"
I20260929 02:12:34.205207 29227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 02:12:34.208686 29242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:34.209887 29227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:12:34.210058 29227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "108165158d11456b9c828ccb9c05bff0"
format_stamp: "Formatted at 2026-09-29 02:12:34 on dist-test-slave-9k0h"
server_key: "35c8e128a9ca9297a5544a27c703a28a"
server_key_iv: "263f6b576c44941ce1e555492be8c19c"
server_key_version: "encryptionkey@0"
I20260929 02:12:34.210304 29227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:34.224910 29227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:34.228565 29227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:34.228853 29227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:34.229753 29227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:34.231069 29227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:34.231170 29227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:34.231316 29227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:34.231395 29227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:34.265614 29227 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:33563
I20260929 02:12:34.265650 29354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:33563 every 8 connection(s)
I20260929 02:12:34.266952 29227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:12:34.269599 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 29227
I20260929 02:12:34.269784 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:12:34.270394 25442 external_mini_cluster.cc:1428] Setting key 1fe2cb0283e0b8bd8f7e600ded2988a0
Sep 29 02:12:34 dist-test-slave-9k0h krb5kdc[28987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647954, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:34.286187 29223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:34.269315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:52179 (local address 127.24.216.190:39397)
0929 02:12:34.269484 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:34.269487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:34.270981 (+ 1494us) server_negotiation.cc:407] Connection header received
0929 02:12:34.272873 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:34.272878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:34.272954 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:34.273062 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:34.274545 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.275213 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.276688 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.277080 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.277212 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:34.280728 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:34.280733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:34.280736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:34.280763 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:34.283350 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:34.284018 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:34.284023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:34.284028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:34.284146 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:34.284562 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:34.284567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:34.284570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:34.284805 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:34.284967 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:34.285610 (+ 643us) server_negotiation.cc:299] Negotiation successful
0929 02:12:34.285834 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:12:34.287160 29357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:34.270009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39397 (local address 127.24.216.130:52179)
0929 02:12:34.270825 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:34.270865 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:34.272619 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:34.273309 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:34.273318 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:34.273769 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:34.274388 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:34.274406 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.275365 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.275376 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:34.276542 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:34.276553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.277290 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.277296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:34.277540 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:34.278284 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:34.278306 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:34.280525 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:34.283477 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:34.283485 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:34.283500 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:34.283897 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:34.284253 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:34.284259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:34.284263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:34.284450 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:34.284924 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:34.284934 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:34.285268 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 02:12:34.285532 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 02:12:34.286046 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":318,"threads_started":1}
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
I20260929 02:12:34.288582 29355 heartbeater.cc:347] Connected to a master server at 127.24.216.190:39397
I20260929 02:12:34.288897 29355 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:34.289623 29355 heartbeater.cc:511] Master 127.24.216.190:39397 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
I20260929 02:12:34.291090 29032 ts_manager.cc:194] Registered new tserver with Master: 108165158d11456b9c828ccb9c05bff0 (127.24.216.130:33563)
I20260929 02:12:34.291873 29032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:52179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:12:34.313418 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39397
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:34975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:34.630445 29360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:34.630712 29360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:34.630806 29360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:34.637974 29360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:34.638137 29360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:34.638342 29360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:12:34.648022 29360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:34975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:34.649394 29360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:34.650843 29360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:34.664038 29368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:34.664139 29366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:34.664121 29365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:34.664745 29360 server_base.cc:1034] running on GCE node
I20260929 02:12:34.665308 29360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:34.666250 29360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:34.667472 29360 hybrid_clock.cc:648] HybridClock initialized: now 1790647954667429 us; error 55 us; skew 500 ppm
I20260929 02:12:34.667776 29360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:34 dist-test-slave-9k0h krb5kdc[28987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647954, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:34.670764 29360 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:12:34.672101 29360 webserver.cc:460] Webserver started at http://127.24.216.131:33795/ using document root <none> and password file <none>
I20260929 02:12:34.672730 29360 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:34.672868 29360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:34.673154 29360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:34.675923 29360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "b9336d2fb0f240abaf627b5f591c1984"
format_stamp: "Formatted at 2026-09-29 02:12:34 on dist-test-slave-9k0h"
server_key: "a5de760f94dd881c10a6c20a4cf49bed"
server_key_iv: "ff69fbbc831a9834ad5328d049b55c87"
server_key_version: "encryptionkey@0"
I20260929 02:12:34.676688 29360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "b9336d2fb0f240abaf627b5f591c1984"
format_stamp: "Formatted at 2026-09-29 02:12:34 on dist-test-slave-9k0h"
server_key: "a5de760f94dd881c10a6c20a4cf49bed"
server_key_iv: "ff69fbbc831a9834ad5328d049b55c87"
server_key_version: "encryptionkey@0"
I20260929 02:12:34.681514 29360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260929 02:12:34.684769 29375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:34.685966 29360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:34.686134 29360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "b9336d2fb0f240abaf627b5f591c1984"
format_stamp: "Formatted at 2026-09-29 02:12:34 on dist-test-slave-9k0h"
server_key: "a5de760f94dd881c10a6c20a4cf49bed"
server_key_iv: "ff69fbbc831a9834ad5328d049b55c87"
server_key_version: "encryptionkey@0"
I20260929 02:12:34.686373 29360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:34.705448 29360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:34.709267 29360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:34.709616 29360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:34.710531 29360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:34.711725 29360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:34.711828 29360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:34.711971 29360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:34.712038 29360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:34.743388 29360 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:44435
I20260929 02:12:34.743426 29487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:44435 every 8 connection(s)
I20260929 02:12:34.744585 29360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:12:34.750499 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 29360
I20260929 02:12:34.750694 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:12:34.751201 25442 external_mini_cluster.cc:1428] Setting key 8ff45c25bef7a2363a8ce82066deb1c7
Sep 29 02:12:34 dist-test-slave-9k0h krb5kdc[28987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647954, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:34.762970 29223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:34.746794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:58857 (local address 127.24.216.190:39397)
0929 02:12:34.746946 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:34.746950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:34.748091 (+ 1141us) server_negotiation.cc:407] Connection header received
0929 02:12:34.749769 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:34.749773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:34.749841 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:34.749929 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:34.751908 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.752727 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.754572 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.754972 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.755063 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:34.758216 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:34.758221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:34.758224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:34.758250 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:34.760498 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:34.761122 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:34.761125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:34.761128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:34.761242 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:34.761578 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:34.761581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:34.761583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:34.761828 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:34.761932 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:34.762566 (+ 634us) server_negotiation.cc:299] Negotiation successful
0929 02:12:34.762720 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:12:34.763959 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:34.747219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39397 (local address 127.24.216.131:58857)
0929 02:12:34.747938 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:34.747979 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:34.749535 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:34.750206 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:34.750220 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:34.750959 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:34.751722 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:34.751756 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.752884 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.752889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:34.754457 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:34.754468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.755144 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.755149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:34.755328 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:34.755969 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:34.755994 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:34.758005 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:34.760614 (+ 2609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:34.760620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:34.760632 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:34.761027 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:34.761332 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:34.761335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:34.761337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:34.761493 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:34.761926 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:34.761931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:34.762285 (+ 354us) client_negotiation.cc:769] Sending connection context
0929 02:12:34.762504 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 02:12:34.762847 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":285,"threads_started":1}
I20260929 02:12:34.765337 29488 heartbeater.cc:347] Connected to a master server at 127.24.216.190:39397
I20260929 02:12:34.765655 29488 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:34.766440 29488 heartbeater.cc:511] Master 127.24.216.190:39397 requested a full tablet report, sending...
I20260929 02:12:34.767727 29031 ts_manager.cc:194] Registered new tserver with Master: b9336d2fb0f240abaf627b5f591c1984 (127.24.216.131:44435)
I20260929 02:12:34.768603 29031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:58857
I20260929 02:12:34.778301 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:12:34 dist-test-slave-9k0h krb5kdc[28987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:12:34 dist-test-slave-9k0h krb5kdc[28987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:12:34.823879 29221 heartbeater.cc:503] Master 127.24.216.190:39397 was elected leader, sending a full tablet report...
I20260929 02:12:34.825100 29495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:34.810969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49316 (local address 127.24.216.129:41031)
0929 02:12:34.811571 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:34.811578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:34.811608 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:12:34.811666 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:34.811672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:34.811880 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:34.811999 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:34.813274 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.814352 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.815639 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.816190 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.816282 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:34.819875 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:34.819885 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:34.819898 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:34.819929 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:34.822667 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:34.823213 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:34.823219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:34.823221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:34.823341 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:34.823623 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:34.823626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:34.823628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:34.823932 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:34.824236 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:34.824550 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 02:12:34.824806 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":316,"threads_started":1}
Sep 29 02:12:34 dist-test-slave-9k0h krb5kdc[28987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:34.841461 29223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:34.828988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.24.216.190:39397)
0929 02:12:34.829179 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:34.829186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:34.829361 (+ 175us) server_negotiation.cc:407] Connection header received
0929 02:12:34.829533 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:34.829539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:34.829602 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:34.829694 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:34.830860 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.831528 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.832703 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.833103 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.833219 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:34.836749 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:34.836753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:34.836756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:34.836790 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:34.839012 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:34.839618 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:34.839631 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:34.839633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:34.839743 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:34.840097 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:34.840103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:34.840107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:34.840400 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:34.840651 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:34.840947 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 02:12:34.841151 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 02:12:34.842073 29031 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40220
I20260929 02:12:34.843000 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 29093
I20260929 02:12:34.853956 29495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:34.843813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.24.216.129:41031)
0929 02:12:34.843958 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:34.843962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:34.844258 (+ 296us) server_negotiation.cc:407] Connection header received
0929 02:12:34.844366 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:34.844370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:34.844423 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:34.844497 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:34.845514 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.846260 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.847372 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.847762 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.847860 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:34.849733 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:34.849737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:34.849740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:34.849766 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:34.852158 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:34.852632 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:34.852635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:34.852637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:34.852737 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:34.853012 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:34.853015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:34.853017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:34.853221 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:34.853374 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:34.853590 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 02:12:34.853728 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 02:12:34.854501 29216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:34.935806 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 29093
I20260929 02:12:34.953359 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 29227
Sep 29 02:12:34 dist-test-slave-9k0h krb5kdc[28987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:12:34.968014 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:34.954226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.24.216.130:33563)
0929 02:12:34.954842 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:34.954850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:34.954894 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:12:34.954968 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:34.954974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:34.955191 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:34.955324 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:34.956398 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.957347 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.958677 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:34.959215 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:34.959314 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:34.963214 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:34.963225 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:34.963239 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:34.963268 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:34.965726 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:34.966369 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:34.966376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:34.966378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:34.966490 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:34.966790 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:34.966793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:34.966795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:34.967095 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:34.967302 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:34.967546 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 02:12:34.967730 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":304,"threads_started":1}
I20260929 02:12:34.969215 29350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:35.049963 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 29227
I20260929 02:12:35.065892 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 29360
Sep 29 02:12:35 dist-test-slave-9k0h krb5kdc[28987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:12:35.080776 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:35.066833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.24.216.131:44435)
0929 02:12:35.067385 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:35.067392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:35.067421 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:12:35.067477 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:35.067484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:35.067713 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:35.067840 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:35.068980 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:35.070015 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:35.071300 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:35.071825 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:35.071917 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:35.075828 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:35.075838 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:35.075851 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:35.075880 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:35.078509 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:35.079077 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:35.079084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:35.079086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:35.079192 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:35.079501 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:35.079504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:35.079506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:35.079807 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:35.080034 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:35.080325 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 02:12:35.080507 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":259,"threads_started":1}
I20260929 02:12:35.081984 29483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:35.162315 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 29360
I20260929 02:12:35.177976 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 29000
I20260929 02:12:35.179121 29062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:35.284157 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 29000
2026-09-29T02:12:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3723 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:12:35 dist-test-slave-9k0h krb5kdc[29507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:12:35 dist-test-slave-9k0h krb5kdc[29507](info): set up 2 sockets
Sep 29 02:12:35 dist-test-slave-9k0h krb5kdc[29507](info): commencing operation
krb5kdc: starting...
W20260929 02:12:36.358683 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:12:36 dist-test-slave-9k0h krb5kdc[29507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:12:36.547735 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:37189
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:37189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:36.855080 29520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:36.855392 29520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:36.855546 29520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:36.862419 29520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:36.862540 29520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:36.862630 29520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:36.862728 29520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:36.862802 29520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:12:36.871908 29520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:37189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:37189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:36.873107 29520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:36.874384 29520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:36.885653 29525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:36.885622 29526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:36.885684 29528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:36.886528 29520 server_base.cc:1034] running on GCE node
I20260929 02:12:36.887259 29520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:36.888607 29520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:36.889830 29520 hybrid_clock.cc:648] HybridClock initialized: now 1790647956889788 us; error 58 us; skew 500 ppm
I20260929 02:12:36.890202 29520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:36 dist-test-slave-9k0h krb5kdc[29507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647956, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:36.893209 29520 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:12:36.894361 29520 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:36.894484 29520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:36.894799 29520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:36.897670 29520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "72d3b30659554b5c80fe84d29bcf45a1"
format_stamp: "Formatted at 2026-09-29 02:12:36 on dist-test-slave-9k0h"
server_key: "42298c687bd6a0c35547324ee0e81a55"
server_key_iv: "d3f5e0e54d2bc9b844e14327888d3125"
server_key_version: "encryptionkey@0"
I20260929 02:12:36.898499 29520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "72d3b30659554b5c80fe84d29bcf45a1"
format_stamp: "Formatted at 2026-09-29 02:12:36 on dist-test-slave-9k0h"
server_key: "42298c687bd6a0c35547324ee0e81a55"
server_key_iv: "d3f5e0e54d2bc9b844e14327888d3125"
server_key_version: "encryptionkey@0"
I20260929 02:12:36.903358 29520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 02:12:36.906719 29534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:36.908051 29520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:12:36.908265 29520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "72d3b30659554b5c80fe84d29bcf45a1"
format_stamp: "Formatted at 2026-09-29 02:12:36 on dist-test-slave-9k0h"
server_key: "42298c687bd6a0c35547324ee0e81a55"
server_key_iv: "d3f5e0e54d2bc9b844e14327888d3125"
server_key_version: "encryptionkey@0"
I20260929 02:12:36.908480 29520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:36.937565 29520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:36.941900 29520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:36.942174 29520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:36.959527 29520 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:37189
I20260929 02:12:36.959553 29585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:37189 every 8 connection(s)
I20260929 02:12:36.960377 29520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:36.963825 29586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:36.965142 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 29520
I20260929 02:12:36.965368 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:36.965989 25442 external_mini_cluster.cc:1428] Setting key 6803a64251fc8ae97f6d1864cac2307f
I20260929 02:12:36.975464 29586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1: Bootstrap starting.
I20260929 02:12:36.979240 29586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1: Neither blocks nor log segments found. Creating new log.
Sep 29 02:12:36 dist-test-slave-9k0h krb5kdc[29507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:36.980374 29586 log.cc:824] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:36.983601 29586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1: No bootstrap required, opened a new log
I20260929 02:12:36.989192 29589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:36.967992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52988 (local address 127.24.216.190:37189)
0929 02:12:36.968815 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:36.968828 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:36.968867 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:12:36.970554 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:36.970577 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:36.970952 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:36.971246 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:36.973034 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:36.974308 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:36.976000 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:36.976597 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:36.976717 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:36.981487 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:36.981494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:36.981507 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:36.981562 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:36.985092 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:36.985686 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:36.985693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:36.985701 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:36.985900 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:36.986252 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:36.986257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:36.986261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:36.986924 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:36.987182 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:36.987491 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 02:12:36.987910 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":382,"threads_started":1}
I20260929 02:12:36.992380 29586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d3b30659554b5c80fe84d29bcf45a1" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37189 } }
I20260929 02:12:36.992666 29586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:36.992733 29586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d3b30659554b5c80fe84d29bcf45a1, State: Initialized, Role: FOLLOWER
I20260929 02:12:36.993184 29586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d3b30659554b5c80fe84d29bcf45a1" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37189 } }
I20260929 02:12:36.993381 29586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:36.993470 29586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:36.993582 29586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:36.995586 29586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d3b30659554b5c80fe84d29bcf45a1" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37189 } }
I20260929 02:12:36.996007 29586 leader_election.cc:302] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72d3b30659554b5c80fe84d29bcf45a1; no voters:
I20260929 02:12:36.996471 29586 leader_election.cc:288] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:36.996572 29591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:36.998054 29591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [term 1 LEADER]: Becoming Leader. State: Replica: 72d3b30659554b5c80fe84d29bcf45a1, State: Running, Role: LEADER
I20260929 02:12:36.998597 29586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:36.998620 29591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d3b30659554b5c80fe84d29bcf45a1" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37189 } }
I20260929 02:12:37.001389 29592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72d3b30659554b5c80fe84d29bcf45a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d3b30659554b5c80fe84d29bcf45a1" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37189 } } }
I20260929 02:12:37.001811 29592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:37.001843 29593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d3b30659554b5c80fe84d29bcf45a1. Latest consensus state: current_term: 1 leader_uuid: "72d3b30659554b5c80fe84d29bcf45a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d3b30659554b5c80fe84d29bcf45a1" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37189 } } }
I20260929 02:12:37.002182 29593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:37.003868 29598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:37.009027 29598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:37.017346 29598 catalog_manager.cc:1347] Generated new cluster ID: 3c19ac36d6a341b281bf11a789acbd11
I20260929 02:12:37.017482 29598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:37.040534 29598 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:37.041796 29598 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:37.047713 29598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1: Generated new TSK 0
I20260929 02:12:37.048434 29598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:12:37.108033 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37189
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:37.436728 29612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:37.436991 29612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:37.437085 29612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:37.444190 29612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:37.444288 29612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:37.444459 29612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:12:37.453927 29612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:37.455200 29612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:37.456429 29612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:37.469313 29617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:37.469538 29618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:37.469722 29620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:37.470548 29612 server_base.cc:1034] running on GCE node
I20260929 02:12:37.471156 29612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:37.471995 29612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:37.473214 29612 hybrid_clock.cc:648] HybridClock initialized: now 1790647957473168 us; error 60 us; skew 500 ppm
I20260929 02:12:37.473510 29612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:37 dist-test-slave-9k0h krb5kdc[29507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647957, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:37.476477 29612 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:12:37.477492 29612 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:37.477617 29612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:37.477936 29612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:37.480734 29612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "513b38dc125c4993b7649e62cd7460ad"
format_stamp: "Formatted at 2026-09-29 02:12:37 on dist-test-slave-9k0h"
server_key: "5553092db0f760255ebd52d24cd5b5b4"
server_key_iv: "db1b3c5afcac23d4f48445177aab7a84"
server_key_version: "encryptionkey@0"
I20260929 02:12:37.481501 29612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "513b38dc125c4993b7649e62cd7460ad"
format_stamp: "Formatted at 2026-09-29 02:12:37 on dist-test-slave-9k0h"
server_key: "5553092db0f760255ebd52d24cd5b5b4"
server_key_iv: "db1b3c5afcac23d4f48445177aab7a84"
server_key_version: "encryptionkey@0"
I20260929 02:12:37.486263 29612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260929 02:12:37.489416 29626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:37.490778 29612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:37.490955 29612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "513b38dc125c4993b7649e62cd7460ad"
format_stamp: "Formatted at 2026-09-29 02:12:37 on dist-test-slave-9k0h"
server_key: "5553092db0f760255ebd52d24cd5b5b4"
server_key_iv: "db1b3c5afcac23d4f48445177aab7a84"
server_key_version: "encryptionkey@0"
I20260929 02:12:37.491163 29612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:37.502492 29612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:37.506268 29612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:37.506518 29612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:37.507350 29612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:37.508575 29612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:37.508684 29612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:37.508810 29612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:37.508904 29612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:37.539605 29612 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:42159
I20260929 02:12:37.539688 29738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:42159 every 8 connection(s)
I20260929 02:12:37.540486 29612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:12:37.550663 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 29612
I20260929 02:12:37.550894 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:12:37.551677 25442 external_mini_cluster.cc:1428] Setting key 7f7923079add4a0f749778f866ff9f9e
Sep 29 02:12:37 dist-test-slave-9k0h krb5kdc[29507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647957, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:37.559311 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:37.542754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:50209 (local address 127.24.216.190:37189)
0929 02:12:37.543183 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:37.543191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:37.544082 (+ 891us) server_negotiation.cc:407] Connection header received
0929 02:12:37.545707 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:37.545711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:37.545778 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:37.545872 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:37.547288 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:37.548055 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:37.549589 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:37.549974 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:37.550061 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:37.553915 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:37.553920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:37.553923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:37.553949 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:37.556360 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:37.557079 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:37.557087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:37.557090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:37.557214 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:37.557609 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:37.557616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:37.557619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:37.557875 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:37.558019 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:37.558730 (+ 711us) server_negotiation.cc:299] Negotiation successful
0929 02:12:37.558955 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":210,"threads_started":1}
I20260929 02:12:37.560246 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:37.543295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:37189 (local address 127.24.216.129:50209)
0929 02:12:37.543930 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:37.543967 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:37.545476 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:37.546055 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:37.546063 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:37.546613 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:37.547125 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:37.547141 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:37.548187 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:37.548191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:37.549451 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:37.549465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:37.550181 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:37.550191 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:37.550381 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:37.551128 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:37.551159 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:37.553723 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:37.556544 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:37.556553 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:37.556568 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:37.556963 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:37.557327 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:37.557332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:37.557336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:37.557502 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:37.558005 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:37.558012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:37.558387 (+ 375us) client_negotiation.cc:769] Sending connection context
0929 02:12:37.558637 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 02:12:37.559108 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":256,"threads_started":1}
I20260929 02:12:37.561684 29739 heartbeater.cc:347] Connected to a master server at 127.24.216.190:37189
I20260929 02:12:37.561960 29739 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:37.562610 29739 heartbeater.cc:511] Master 127.24.216.190:37189 requested a full tablet report, sending...
I20260929 02:12:37.564661 29551 ts_manager.cc:194] Registered new tserver with Master: 513b38dc125c4993b7649e62cd7460ad (127.24.216.129:42159)
I20260929 02:12:37.566771 29551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:50209
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:12:37.594363 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37189
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:37.905655 29745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:37.905921 29745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:37.906015 29745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:37.913193 29745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:37.913301 29745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:37.913431 29745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:12:37.923079 29745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:37.924256 29745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:37.925663 29745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:37.938297 29753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:37.938386 29750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:37.938417 29751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:37.939148 29745 server_base.cc:1034] running on GCE node
I20260929 02:12:37.939662 29745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:37.940565 29745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:37.941754 29745 hybrid_clock.cc:648] HybridClock initialized: now 1790647957941694 us; error 51 us; skew 500 ppm
I20260929 02:12:37.942046 29745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:37 dist-test-slave-9k0h krb5kdc[29507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647957, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:37.944962 29745 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:12:37.946192 29745 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:37.946291 29745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:37.946647 29745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:37.949493 29745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "fd445102f44c4b99bd5c83f7b6c0c13e"
format_stamp: "Formatted at 2026-09-29 02:12:37 on dist-test-slave-9k0h"
server_key: "c899ead3f74c379f9687590480b6454a"
server_key_iv: "2d74cc9c17bb492da73386d037cad1fe"
server_key_version: "encryptionkey@0"
I20260929 02:12:37.950328 29745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "fd445102f44c4b99bd5c83f7b6c0c13e"
format_stamp: "Formatted at 2026-09-29 02:12:37 on dist-test-slave-9k0h"
server_key: "c899ead3f74c379f9687590480b6454a"
server_key_iv: "2d74cc9c17bb492da73386d037cad1fe"
server_key_version: "encryptionkey@0"
I20260929 02:12:37.955053 29745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:12:37.958186 29759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:37.959501 29745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:12:37.959661 29745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "fd445102f44c4b99bd5c83f7b6c0c13e"
format_stamp: "Formatted at 2026-09-29 02:12:37 on dist-test-slave-9k0h"
server_key: "c899ead3f74c379f9687590480b6454a"
server_key_iv: "2d74cc9c17bb492da73386d037cad1fe"
server_key_version: "encryptionkey@0"
I20260929 02:12:37.959890 29745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:37.980695 29745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:37.984604 29745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:37.984884 29745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:37.985810 29745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:37.987159 29745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:37.987282 29745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:37.987416 29745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:37.987502 29745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:38.019989 29745 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:34005
I20260929 02:12:38.020074 29871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:34005 every 8 connection(s)
I20260929 02:12:38.020874 29745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:12:38.021689 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 29745
I20260929 02:12:38.021883 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:12:38.022449 25442 external_mini_cluster.cc:1428] Setting key e2b3c0f9dd661db5bcad732eaa9c6f60
Sep 29 02:12:38 dist-test-slave-9k0h krb5kdc[29507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647957, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
I20260929 02:12:38.042235 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.023578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:57333 (local address 127.24.216.190:37189)
0929 02:12:38.023881 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:38.023891 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:38.025315 (+ 1424us) server_negotiation.cc:407] Connection header received
0929 02:12:38.027154 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:38.027159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:38.027246 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:38.027362 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:38.028804 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.029501 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.032126 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.032533 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.032634 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.036704 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:38.036709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:38.036712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:38.036740 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:38.039308 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:38.040021 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:38.040033 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:38.040036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:38.040148 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:38.040556 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:38.040562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:38.040566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:38.040828 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:38.040983 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:38.041634 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 02:12:38.041863 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 02:12:38.043422 29874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.024223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:37189 (local address 127.24.216.130:57333)
0929 02:12:38.025165 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:38.025203 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:38.026903 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:38.027579 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:38.027588 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:38.028091 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:38.028628 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:38.028645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.029656 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.029668 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:38.031959 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:38.031981 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.032748 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.032758 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:38.033073 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.033795 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:38.033826 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:38.036500 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:38.039468 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:38.039476 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:38.039491 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:38.039899 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:38.040255 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:38.040261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:38.040265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:38.040439 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:38.040960 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:38.040968 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:38.041286 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 02:12:38.041541 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 02:12:38.042079 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":310,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
I20260929 02:12:38.044984 29872 heartbeater.cc:347] Connected to a master server at 127.24.216.190:37189
I20260929 02:12:38.045300 29872 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:38.045919 29872 heartbeater.cc:511] Master 127.24.216.190:37189 requested a full tablet report, sending...
I20260929 02:12:38.047468 29551 ts_manager.cc:194] Registered new tserver with Master: fd445102f44c4b99bd5c83f7b6c0c13e (127.24.216.130:34005)
I20260929 02:12:38.048717 29551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:57333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:12:38.067643 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37189
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:38.385205 29877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:38.385536 29877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:38.385705 29877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:38.392617 29877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:38.392746 29877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:38.392915 29877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:12:38.402406 29877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:38.403640 29877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:38.404980 29877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:38.418622 29882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:38.418547 29883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:38.418588 29885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:38.419596 29877 server_base.cc:1034] running on GCE node
I20260929 02:12:38.420150 29877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:38.421048 29877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:38.422245 29877 hybrid_clock.cc:648] HybridClock initialized: now 1790647958422175 us; error 54 us; skew 500 ppm
I20260929 02:12:38.422503 29877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:38 dist-test-slave-9k0h krb5kdc[29507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647958, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:38.431888 29877 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:12:38.433001 29877 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:38.433135 29877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:38.433439 29877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:38.436419 29877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "adb17a51a0ad4ede9cec06d518d58ffa"
format_stamp: "Formatted at 2026-09-29 02:12:38 on dist-test-slave-9k0h"
server_key: "d59d1e45a64113e835cb9ad6907e475d"
server_key_iv: "6bc299fc6bf155c6d5fcc3b58a6c6daa"
server_key_version: "encryptionkey@0"
I20260929 02:12:38.437201 29877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "adb17a51a0ad4ede9cec06d518d58ffa"
format_stamp: "Formatted at 2026-09-29 02:12:38 on dist-test-slave-9k0h"
server_key: "d59d1e45a64113e835cb9ad6907e475d"
server_key_iv: "6bc299fc6bf155c6d5fcc3b58a6c6daa"
server_key_version: "encryptionkey@0"
I20260929 02:12:38.442224 29877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 02:12:38.445438 29891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:38.446709 29877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:12:38.446883 29877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "adb17a51a0ad4ede9cec06d518d58ffa"
format_stamp: "Formatted at 2026-09-29 02:12:38 on dist-test-slave-9k0h"
server_key: "d59d1e45a64113e835cb9ad6907e475d"
server_key_iv: "6bc299fc6bf155c6d5fcc3b58a6c6daa"
server_key_version: "encryptionkey@0"
I20260929 02:12:38.447085 29877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:38.463070 29877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:38.466996 29877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:38.467252 29877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:38.468133 29877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:38.469388 29877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:38.469509 29877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:38.469630 29877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:38.469726 29877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:38.501876 29877 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:42391
I20260929 02:12:38.501929 30003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:42391 every 8 connection(s)
I20260929 02:12:38.502853 29877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:12:38.505352 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 29877
I20260929 02:12:38.505556 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:12:38.506033 25442 external_mini_cluster.cc:1428] Setting key ffb7346f8c6b39c21fe1b0fcba546d77
Sep 29 02:12:38 dist-test-slave-9k0h krb5kdc[29507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647958, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:38.522641 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.505471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:49655 (local address 127.24.216.190:37189)
0929 02:12:38.505642 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:38.505648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:38.507210 (+ 1562us) server_negotiation.cc:407] Connection header received
0929 02:12:38.509454 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:38.509459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:38.509546 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:38.509650 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:38.511162 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.511934 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.513564 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.513997 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.514121 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.517283 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:38.517290 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:38.517299 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:38.517327 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:38.519770 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:38.520469 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:38.520481 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:38.520484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:38.520597 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:38.521001 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:38.521005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:38.521029 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:38.521278 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:38.521434 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:38.522060 (+ 626us) server_negotiation.cc:299] Negotiation successful
0929 02:12:38.522293 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 02:12:38.523495 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.506100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:37189 (local address 127.24.216.131:49655)
0929 02:12:38.507060 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:38.507107 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:38.509162 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:38.509877 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:38.509885 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:38.510471 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:38.511006 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:38.511026 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.512173 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.512178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:38.513436 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:38.513447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.514241 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.514249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:38.514481 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.515054 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:38.515073 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:38.517109 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:38.519925 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:38.519933 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:38.519948 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:38.520344 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:38.520704 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:38.520710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:38.520713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:38.520882 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:38.521408 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:38.521415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:38.521726 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 02:12:38.521974 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 02:12:38.522497 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":409,"threads_started":1}
I20260929 02:12:38.524658 30004 heartbeater.cc:347] Connected to a master server at 127.24.216.190:37189
I20260929 02:12:38.524945 30004 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:38.525591 30004 heartbeater.cc:511] Master 127.24.216.190:37189 requested a full tablet report, sending...
I20260929 02:12:38.527030 29551 ts_manager.cc:194] Registered new tserver with Master: adb17a51a0ad4ede9cec06d518d58ffa (127.24.216.131:42391)
I20260929 02:12:38.527799 29551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:49655
I20260929 02:12:38.532871 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:12:38.549954 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.537154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.24.216.190:37189)
0929 02:12:38.537341 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:38.537349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:38.537819 (+ 470us) server_negotiation.cc:407] Connection header received
0929 02:12:38.537967 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:38.537970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:38.538023 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:38.538093 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:38.539641 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.540299 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.542023 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.542557 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.542653 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.545319 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:38.545323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:38.545327 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:38.545353 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:38.547855 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:38.548418 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:38.548422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:38.548424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:38.548534 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:38.548814 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:38.548817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:38.548818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:38.549068 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:38.549232 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:38.549486 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 02:12:38.549623 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:12:38.553481 29551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:12:38.559921 29551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:12:38.569633 29739 heartbeater.cc:503] Master 127.24.216.190:37189 was elected leader, sending a full tablet report...
I20260929 02:12:38.582880 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.571193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.24.216.129:42159)
0929 02:12:38.574036 (+ 2843us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:38.574046 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:38.574072 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:12:38.574185 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:38.574195 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:38.574405 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:38.574588 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:38.575462 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.576956 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.579899 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.581284 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.581463 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.581612 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:38.582431 (+ 819us) server_negotiation.cc:299] Negotiation successful
0929 02:12:38.582571 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2684,"thread_start_us":2543,"threads_started":1}
I20260929 02:12:38.582947 30016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.571436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:42391 (local address 127.0.0.1:59202)
0929 02:12:38.573358 (+ 1922us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:38.573371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:38.573517 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:38.574423 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:38.574434 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:38.574462 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:38.574850 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:38.574863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.576952 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.576956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:38.579710 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:38.579724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.581787 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.581791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:38.582040 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.582185 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 02:12:38.582304 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 02:12:38.582482 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1828,"mutex_wait_us":203,"thread_start_us":341,"threads_started":1}
I20260929 02:12:38.583626 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.571647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59202 (local address 127.24.216.131:42391)
0929 02:12:38.573070 (+ 1423us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:38.573076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:38.573477 (+ 401us) server_negotiation.cc:407] Connection header received
0929 02:12:38.573684 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:38.573694 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:38.573936 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:38.574118 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:38.575100 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.576787 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.579889 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.581285 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.581455 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.581617 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:38.583142 (+ 1525us) server_negotiation.cc:299] Negotiation successful
0929 02:12:38.583303 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1294,"thread_start_us":1173,"threads_started":1}
I20260929 02:12:38.584872 30018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.571075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:42159 (local address 127.0.0.1:33738)
0929 02:12:38.572139 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:38.572179 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:38.572335 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:38.574808 (+ 2473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:38.574813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:38.574841 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:38.575269 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:38.575279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.577111 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.577115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:38.579716 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:38.579724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.581762 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.581773 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:38.582040 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.582179 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 02:12:38.582303 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 02:12:38.583552 (+ 1249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":1227,"thread_start_us":2096,"threads_started":1}
I20260929 02:12:38.587390 29939 tablet_service.cc:1467] Processing CreateTablet for tablet 56634c2f87614627afb6d9e302f03b9d (DEFAULT_TABLE table=test-table [id=e4733b19e1e24b54bb11935254fc6627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:38.588269 29674 tablet_service.cc:1467] Processing CreateTablet for tablet 56634c2f87614627afb6d9e302f03b9d (DEFAULT_TABLE table=test-table [id=e4733b19e1e24b54bb11935254fc6627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:38.588878 29939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56634c2f87614627afb6d9e302f03b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:38.589416 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.574358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:34005 (local address 127.0.0.1:38738)
0929 02:12:38.582819 (+ 8461us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:38.582835 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:38.582956 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:38.583593 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:38.583597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:38.583619 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:38.583961 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:38.583976 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.585901 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.585907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:38.587399 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:38.587410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.588662 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.588668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:38.588820 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.588864 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 02:12:38.588951 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 02:12:38.589063 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8327,"thread_start_us":8178,"threads_started":1}
I20260929 02:12:38.589511 29674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56634c2f87614627afb6d9e302f03b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:38.590119 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.573903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38738 (local address 127.24.216.130:34005)
0929 02:12:38.582468 (+ 8565us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:38.582478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:38.583081 (+ 603us) server_negotiation.cc:407] Connection header received
0929 02:12:38.583184 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:38.583193 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:38.583407 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:38.584161 (+ 754us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:38.584351 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.585754 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.587542 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.588469 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.588647 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.588810 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:38.589700 (+ 890us) server_negotiation.cc:299] Negotiation successful
0929 02:12:38.589857 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8427,"thread_start_us":8342,"threads_started":1}
I20260929 02:12:38.592581 29807 tablet_service.cc:1467] Processing CreateTablet for tablet 56634c2f87614627afb6d9e302f03b9d (DEFAULT_TABLE table=test-table [id=e4733b19e1e24b54bb11935254fc6627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:38.593748 29807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56634c2f87614627afb6d9e302f03b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:38.603256 30022 tablet_bootstrap.cc:492] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa: Bootstrap starting.
I20260929 02:12:38.605319 30023 tablet_bootstrap.cc:492] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad: Bootstrap starting.
I20260929 02:12:38.608656 30022 tablet_bootstrap.cc:654] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:38.609416 30023 tablet_bootstrap.cc:654] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:38.609452 30024 tablet_bootstrap.cc:492] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e: Bootstrap starting.
I20260929 02:12:38.609937 30022 log.cc:824] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:38.610626 30023 log.cc:824] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:38.613255 30022 tablet_bootstrap.cc:492] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa: No bootstrap required, opened a new log
I20260929 02:12:38.613236 30024 tablet_bootstrap.cc:654] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:38.613526 30022 ts_tablet_manager.cc:1397] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260929 02:12:38.614457 30024 log.cc:824] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:38.617570 30024 tablet_bootstrap.cc:492] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e: No bootstrap required, opened a new log
I20260929 02:12:38.617835 30024 ts_tablet_manager.cc:1397] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 02:12:38.618391 30023 tablet_bootstrap.cc:492] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad: No bootstrap required, opened a new log
I20260929 02:12:38.618641 30023 ts_tablet_manager.cc:1397] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20260929 02:12:38.622084 30022 raft_consensus.cc:348] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513b38dc125c4993b7649e62cd7460ad" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42159 } } peers { permanent_uuid: "adb17a51a0ad4ede9cec06d518d58ffa" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 42391 } } peers { permanent_uuid: "fd445102f44c4b99bd5c83f7b6c0c13e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 34005 } }
I20260929 02:12:38.622511 30022 raft_consensus.cc:374] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:38.622640 30022 raft_consensus.cc:729] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb17a51a0ad4ede9cec06d518d58ffa, State: Initialized, Role: FOLLOWER
I20260929 02:12:38.623227 30022 consensus_queue.cc:260] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513b38dc125c4993b7649e62cd7460ad" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42159 } } peers { permanent_uuid: "adb17a51a0ad4ede9cec06d518d58ffa" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 42391 } } peers { permanent_uuid: "fd445102f44c4b99bd5c83f7b6c0c13e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 34005 } }
I20260929 02:12:38.624287 30004 heartbeater.cc:503] Master 127.24.216.190:37189 was elected leader, sending a full tablet report...
I20260929 02:12:38.624717 30022 ts_tablet_manager.cc:1428] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 02:12:38.628724 30024 raft_consensus.cc:348] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513b38dc125c4993b7649e62cd7460ad" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42159 } } peers { permanent_uuid: "adb17a51a0ad4ede9cec06d518d58ffa" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 42391 } } peers { permanent_uuid: "fd445102f44c4b99bd5c83f7b6c0c13e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 34005 } }
I20260929 02:12:38.628719 30023 raft_consensus.cc:348] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513b38dc125c4993b7649e62cd7460ad" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42159 } } peers { permanent_uuid: "adb17a51a0ad4ede9cec06d518d58ffa" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 42391 } } peers { permanent_uuid: "fd445102f44c4b99bd5c83f7b6c0c13e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 34005 } }
I20260929 02:12:38.629145 30024 raft_consensus.cc:374] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:38.629145 30023 raft_consensus.cc:374] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:38.629240 30023 raft_consensus.cc:729] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513b38dc125c4993b7649e62cd7460ad, State: Initialized, Role: FOLLOWER
I20260929 02:12:38.629269 30024 raft_consensus.cc:729] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd445102f44c4b99bd5c83f7b6c0c13e, State: Initialized, Role: FOLLOWER
I20260929 02:12:38.629856 30023 consensus_queue.cc:260] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513b38dc125c4993b7649e62cd7460ad" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42159 } } peers { permanent_uuid: "adb17a51a0ad4ede9cec06d518d58ffa" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 42391 } } peers { permanent_uuid: "fd445102f44c4b99bd5c83f7b6c0c13e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 34005 } }
I20260929 02:12:38.629856 30024 consensus_queue.cc:260] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513b38dc125c4993b7649e62cd7460ad" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42159 } } peers { permanent_uuid: "adb17a51a0ad4ede9cec06d518d58ffa" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 42391 } } peers { permanent_uuid: "fd445102f44c4b99bd5c83f7b6c0c13e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 34005 } }
I20260929 02:12:38.631002 29872 heartbeater.cc:503] Master 127.24.216.190:37189 was elected leader, sending a full tablet report...
I20260929 02:12:38.633253 30024 ts_tablet_manager.cc:1428] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e: Time spent starting tablet: real 0.015s user 0.002s sys 0.010s
I20260929 02:12:38.634843 30023 ts_tablet_manager.cc:1428] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260929 02:12:38.642119 30030 raft_consensus.cc:482] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:12:38.642397 30030 raft_consensus.cc:504] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513b38dc125c4993b7649e62cd7460ad" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42159 } } peers { permanent_uuid: "adb17a51a0ad4ede9cec06d518d58ffa" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 42391 } } peers { permanent_uuid: "fd445102f44c4b99bd5c83f7b6c0c13e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 34005 } }
I20260929 02:12:38.643980 30030 leader_election.cc:288] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adb17a51a0ad4ede9cec06d518d58ffa (127.24.216.131:42391), fd445102f44c4b99bd5c83f7b6c0c13e (127.24.216.130:34005)
I20260929 02:12:38.652530 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.644348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:54579 (local address 127.24.216.131:42391)
0929 02:12:38.644577 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:38.644586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:38.645181 (+ 595us) server_negotiation.cc:407] Connection header received
0929 02:12:38.645309 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:38.645314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:38.645384 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:38.645463 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:38.646151 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.647214 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.650318 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.651609 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.651737 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.651830 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:38.652176 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 02:12:38.652292 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:12:38.652530 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.644450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:38867 (local address 127.24.216.130:34005)
0929 02:12:38.644651 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:38.644658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:38.645006 (+ 348us) server_negotiation.cc:407] Connection header received
0929 02:12:38.645199 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:38.645206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:38.645304 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:38.645397 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:38.646155 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.647214 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.650330 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.651518 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.651681 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.651752 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:38.652176 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 02:12:38.652300 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:12:38.652734 30031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.644228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:34005 (local address 127.24.216.129:38867)
0929 02:12:38.644878 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:38.644906 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:38.645052 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:38.645412 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:38.645415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:38.645457 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:38.646024 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:38.646035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.647344 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.647350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:38.650174 (+ 2824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:38.650188 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.651763 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.651772 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:38.652002 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.652048 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 02:12:38.652139 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 02:12:38.652313 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":316,"thread_start_us":380,"threads_started":1}
I20260929 02:12:38.652875 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.644286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:42391 (local address 127.24.216.129:54579)
0929 02:12:38.645092 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:38.645108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:38.645209 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:38.645481 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:38.645488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:38.645511 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:38.645964 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:38.645982 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.647346 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.647350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:38.650188 (+ 2838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:38.650198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.651806 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.651812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:38.651995 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.652044 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 02:12:38.652139 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 02:12:38.652313 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":266,"thread_start_us":527,"threads_started":1}
I20260929 02:12:38.653641 29827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56634c2f87614627afb6d9e302f03b9d" candidate_uuid: "513b38dc125c4993b7649e62cd7460ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd445102f44c4b99bd5c83f7b6c0c13e" is_pre_election: true
I20260929 02:12:38.653677 29959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56634c2f87614627afb6d9e302f03b9d" candidate_uuid: "513b38dc125c4993b7649e62cd7460ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb17a51a0ad4ede9cec06d518d58ffa" is_pre_election: true
I20260929 02:12:38.654049 29959 raft_consensus.cc:2454] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 513b38dc125c4993b7649e62cd7460ad in term 0.
I20260929 02:12:38.654062 29827 raft_consensus.cc:2454] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 513b38dc125c4993b7649e62cd7460ad in term 0.
I20260929 02:12:38.654676 29629 leader_election.cc:302] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 513b38dc125c4993b7649e62cd7460ad, adb17a51a0ad4ede9cec06d518d58ffa, fd445102f44c4b99bd5c83f7b6c0c13e; no voters:
I20260929 02:12:38.654963 30030 raft_consensus.cc:2781] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:12:38.655097 30030 raft_consensus.cc:482] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:12:38.655218 30030 raft_consensus.cc:3037] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:38.657218 30030 raft_consensus.cc:504] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513b38dc125c4993b7649e62cd7460ad" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42159 } } peers { permanent_uuid: "adb17a51a0ad4ede9cec06d518d58ffa" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 42391 } } peers { permanent_uuid: "fd445102f44c4b99bd5c83f7b6c0c13e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 34005 } }
I20260929 02:12:38.657742 30030 leader_election.cc:288] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [CANDIDATE]: Term 1 election: Requested vote from peers adb17a51a0ad4ede9cec06d518d58ffa (127.24.216.131:42391), fd445102f44c4b99bd5c83f7b6c0c13e (127.24.216.130:34005)
I20260929 02:12:38.658239 29959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56634c2f87614627afb6d9e302f03b9d" candidate_uuid: "513b38dc125c4993b7649e62cd7460ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb17a51a0ad4ede9cec06d518d58ffa"
I20260929 02:12:38.658257 29827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56634c2f87614627afb6d9e302f03b9d" candidate_uuid: "513b38dc125c4993b7649e62cd7460ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd445102f44c4b99bd5c83f7b6c0c13e"
I20260929 02:12:38.658434 29827 raft_consensus.cc:3037] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:38.658425 29959 raft_consensus.cc:3037] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:38.660854 29827 raft_consensus.cc:2454] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 513b38dc125c4993b7649e62cd7460ad in term 1.
I20260929 02:12:38.660866 29959 raft_consensus.cc:2454] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 513b38dc125c4993b7649e62cd7460ad in term 1.
I20260929 02:12:38.661211 29627 leader_election.cc:302] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 513b38dc125c4993b7649e62cd7460ad, adb17a51a0ad4ede9cec06d518d58ffa; no voters:
I20260929 02:12:38.661507 30030 raft_consensus.cc:2781] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:38.661960 30030 raft_consensus.cc:686] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [term 1 LEADER]: Becoming Leader. State: Replica: 513b38dc125c4993b7649e62cd7460ad, State: Running, Role: LEADER
I20260929 02:12:38.662436 30030 consensus_queue.cc:237] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513b38dc125c4993b7649e62cd7460ad" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42159 } } peers { permanent_uuid: "adb17a51a0ad4ede9cec06d518d58ffa" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 42391 } } peers { permanent_uuid: "fd445102f44c4b99bd5c83f7b6c0c13e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 34005 } }
I20260929 02:12:38.666838 29551 catalog_manager.cc:5495] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 513b38dc125c4993b7649e62cd7460ad (127.24.216.129). New cstate: current_term: 1 leader_uuid: "513b38dc125c4993b7649e62cd7460ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513b38dc125c4993b7649e62cd7460ad" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adb17a51a0ad4ede9cec06d518d58ffa" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 42391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd445102f44c4b99bd5c83f7b6c0c13e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 34005 } health_report { overall_health: UNKNOWN } } }
I20260929 02:12:38.689631 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.684637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.24.216.129:42159)
0929 02:12:38.684831 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:38.684838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:38.684855 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:12:38.684934 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:38.684937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:38.685004 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:38.685149 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:12:38.685496 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.686248 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.687569 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.687995 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.688125 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.688584 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:12:38.689067 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:12:38.689232 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:38.689284 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 02:12:38.689373 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:12:38.697290 29827 raft_consensus.cc:1261] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e [term 1 FOLLOWER]: Refusing update from remote peer 513b38dc125c4993b7649e62cd7460ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:38.697410 29959 raft_consensus.cc:1261] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa [term 1 FOLLOWER]: Refusing update from remote peer 513b38dc125c4993b7649e62cd7460ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:38.698343 30030 consensus_queue.cc:1035] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd445102f44c4b99bd5c83f7b6c0c13e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 34005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:38.698669 30033 consensus_queue.cc:1035] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "adb17a51a0ad4ede9cec06d518d58ffa" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 42391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:38.709697 30037 mvcc.cc:205] Tried to move back new op lower bound from 7334494038815162368 to 7334494038684778496. Current Snapshot: MvccSnapshot[applied={T|T < 7334494038815162368}]
I20260929 02:12:38.710203 30036 mvcc.cc:205] Tried to move back new op lower bound from 7334494038815162368 to 7334494038684778496. Current Snapshot: MvccSnapshot[applied={T|T < 7334494038815162368}]
I20260929 02:12:38.723235 29551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:38.723452 29551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:38.726824 29551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72d3b30659554b5c80fe84d29bcf45a1: Sending DeleteTablet for 3 replicas of tablet 56634c2f87614627afb6d9e302f03b9d
I20260929 02:12:38.727871 29939 tablet_service.cc:1514] Processing DeleteTablet for tablet 56634c2f87614627afb6d9e302f03b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:38 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:59202
I20260929 02:12:38.727890 29674 tablet_service.cc:1514] Processing DeleteTablet for tablet 56634c2f87614627afb6d9e302f03b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:38 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:33738
I20260929 02:12:38.728749 29807 tablet_service.cc:1514] Processing DeleteTablet for tablet 56634c2f87614627afb6d9e302f03b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:38 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:38738
I20260929 02:12:38.729017 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 29612
I20260929 02:12:38.740265 30045 tablet_replica.cc:330] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad: stopping tablet replica
I20260929 02:12:38.740839 30046 tablet_replica.cc:330] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa: stopping tablet replica
I20260929 02:12:38.741529 30046 raft_consensus.cc:2229] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:38.742110 30045 raft_consensus.cc:2229] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:12:38.742834 30045 raft_consensus.cc:2258] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:38.743408 30046 raft_consensus.cc:2258] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:38.744053 30047 tablet_replica.cc:330] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e: stopping tablet replica
I20260929 02:12:38.744478 30047 raft_consensus.cc:2229] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:38.744997 30047 raft_consensus.cc:2258] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:38.745586 30046 ts_tablet_manager.cc:1905] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:38.747280 30047 ts_tablet_manager.cc:1905] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:38.751617 30045 ts_tablet_manager.cc:1905] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:12:38 dist-test-slave-9k0h krb5kdc[29507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:12:38.754343 30047 ts_tablet_manager.cc:1918] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:12:38.754594 30047 log.cc:1196] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/56634c2f87614627afb6d9e302f03b9d
I20260929 02:12:38.755200 30047 ts_tablet_manager.cc:1939] T 56634c2f87614627afb6d9e302f03b9d P fd445102f44c4b99bd5c83f7b6c0c13e: Deleting consensus metadata
I20260929 02:12:38.757135 29537 catalog_manager.cc:4841] TS fd445102f44c4b99bd5c83f7b6c0c13e (127.24.216.130:34005): tablet 56634c2f87614627afb6d9e302f03b9d (table test-table [id=e4733b19e1e24b54bb11935254fc6627]) successfully deleted
I20260929 02:12:38.759389 30045 ts_tablet_manager.cc:1918] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:12:38.759614 30045 log.cc:1196] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/56634c2f87614627afb6d9e302f03b9d
I20260929 02:12:38.760236 30045 ts_tablet_manager.cc:1939] T 56634c2f87614627afb6d9e302f03b9d P 513b38dc125c4993b7649e62cd7460ad: Deleting consensus metadata
I20260929 02:12:38.760659 30046 ts_tablet_manager.cc:1918] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:12:38.760900 30046 log.cc:1196] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/56634c2f87614627afb6d9e302f03b9d
I20260929 02:12:38.761487 30046 ts_tablet_manager.cc:1939] T 56634c2f87614627afb6d9e302f03b9d P adb17a51a0ad4ede9cec06d518d58ffa: Deleting consensus metadata
I20260929 02:12:38.761852 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.730596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.24.216.129:42159)
0929 02:12:38.730817 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:38.730826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:38.740583 (+ 9757us) server_negotiation.cc:407] Connection header received
0929 02:12:38.741120 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:38.741128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:38.741242 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:38.741344 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:38.744591 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.745602 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.748061 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.748742 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.748848 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.754231 (+ 5383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:38.754245 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:38.754267 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:38.754328 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:38.758316 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:38.759142 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:38.759154 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:38.759157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:38.759336 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:38.759786 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:38.759791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:38.759794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:38.760196 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:38.760504 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:38.761112 (+ 608us) server_negotiation.cc:299] Negotiation successful
0929 02:12:38.761424 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 02:12:38.762123 29537 catalog_manager.cc:4841] TS 513b38dc125c4993b7649e62cd7460ad (127.24.216.129:42159): tablet 56634c2f87614627afb6d9e302f03b9d (table test-table [id=e4733b19e1e24b54bb11935254fc6627]) successfully deleted
I20260929 02:12:38.763455 29535 catalog_manager.cc:4841] TS adb17a51a0ad4ede9cec06d518d58ffa (127.24.216.131:42391): tablet 56634c2f87614627afb6d9e302f03b9d (table test-table [id=e4733b19e1e24b54bb11935254fc6627]) successfully deleted
I20260929 02:12:38.763576 29734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:38.859401 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 29612
I20260929 02:12:38.877703 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 29745
Sep 29 02:12:38 dist-test-slave-9k0h krb5kdc[29507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:12:38.891472 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.878480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.24.216.130:34005)
0929 02:12:38.878650 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:38.878657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:38.878750 (+ 93us) server_negotiation.cc:407] Connection header received
0929 02:12:38.878960 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:38.878964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:38.879025 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:38.879137 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:38.880324 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.881052 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.882296 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:38.882683 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:38.882781 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:38.886732 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:38.886742 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:38.886757 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:38.886788 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:38.889268 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:38.889799 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:38.889805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:38.889807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:38.889911 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:38.890332 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:38.890335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:38.890337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:38.890649 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:38.890853 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:38.891056 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 02:12:38.891205 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:12:38.892081 29867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:38.982999 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 29745
I20260929 02:12:38.999126 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 29877
Sep 29 02:12:39 dist-test-slave-9k0h krb5kdc[29507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:12:39.013336 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:38.999979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.24.216.131:42391)
0929 02:12:39.000163 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:39.000168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:39.000204 (+ 36us) server_negotiation.cc:407] Connection header received
0929 02:12:39.000289 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:39.000293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:39.000353 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:39.000434 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:39.001597 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:39.002351 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:39.003478 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:39.003803 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:39.003888 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:39.007923 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:39.007937 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:39.007953 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:39.007984 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:39.010766 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:39.011508 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:39.011515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:39.011517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:39.011634 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:39.011972 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:39.011975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:39.011977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:39.012371 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:39.012565 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:39.012857 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 02:12:39.013003 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:12:39.014037 29999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:39.100826 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 29877
I20260929 02:12:39.117393 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 29520
I20260929 02:12:39.118534 29581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:39.225560 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 29520
2026-09-29T02:12:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3941 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T02:12:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:39Z Disabled control of system clock
I20260929 02:12:39.291875 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:46185
--webserver_interface=127.24.216.190
--webserver_port=0
--builtin_ntp_servers=127.24.216.148:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:46185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 02:12:39.602949 30056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:39.603228 30056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:39.603380 30056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:39.610397 30056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:39.610489 30056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:39.610590 30056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:39.610705 30056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:39.610790 30056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:12:39.619657 30056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:46185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:46185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:39.620945 30056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:39.622596 30056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:39.634182 30062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:39.634123 30064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:39.634218 30061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:39.635064 30056 server_base.cc:1034] running on GCE node
I20260929 02:12:39.635828 30056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:39.637202 30056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:39.638428 30056 hybrid_clock.cc:648] HybridClock initialized: now 1790647959638372 us; error 47 us; skew 500 ppm
I20260929 02:12:39.638752 30056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:12:39.640895 30056 webserver.cc:460] Webserver started at http://127.24.216.190:40897/ using document root <none> and password file <none>
I20260929 02:12:39.641557 30056 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:39.641685 30056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:39.642009 30056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:39.644815 30056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "e48156d6e37745f5835282c358bdddad"
format_stamp: "Formatted at 2026-09-29 02:12:39 on dist-test-slave-9k0h"
server_key: "e30b2504fd280f6f761ddd3be40a38f1"
server_key_iv: "a8e8bf7822911c8b9f8bcc7fee6c45b1"
server_key_version: "encryptionkey@0"
I20260929 02:12:39.645586 30056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "e48156d6e37745f5835282c358bdddad"
format_stamp: "Formatted at 2026-09-29 02:12:39 on dist-test-slave-9k0h"
server_key: "e30b2504fd280f6f761ddd3be40a38f1"
server_key_iv: "a8e8bf7822911c8b9f8bcc7fee6c45b1"
server_key_version: "encryptionkey@0"
I20260929 02:12:39.650476 30056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:12:39.653609 30070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:39.654901 30056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:39.655071 30056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "e48156d6e37745f5835282c358bdddad"
format_stamp: "Formatted at 2026-09-29 02:12:39 on dist-test-slave-9k0h"
server_key: "e30b2504fd280f6f761ddd3be40a38f1"
server_key_iv: "a8e8bf7822911c8b9f8bcc7fee6c45b1"
server_key_version: "encryptionkey@0"
I20260929 02:12:39.655269 30056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:39.660827 30056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:39.661518 30056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:39.661796 30056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:39.678937 30056 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:46185
I20260929 02:12:39.678973 30121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:46185 every 8 connection(s)
I20260929 02:12:39.680191 30056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:39.683727 30122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:39.687688 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 30056
I20260929 02:12:39.687870 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:39.688494 25442 external_mini_cluster.cc:1428] Setting key c9210f2ed70225455c37f711ce2012db
I20260929 02:12:39.693909 30122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad: Bootstrap starting.
I20260929 02:12:39.696254 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:39.690415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.24.216.190:46185)
0929 02:12:39.691408 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:39.691417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:39.691458 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:12:39.693030 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:39.693049 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:39.693416 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:39.693711 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:39.694809 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:39.694815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:39.694827 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:39.694839 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:39.694889 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:39.694903 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:39.695057 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:39.695170 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 02:12:39.695419 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":350,"threads_started":1}
I20260929 02:12:39.697190 30122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:39.698235 30122 log.cc:824] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:39.700704 30122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad: No bootstrap required, opened a new log
I20260929 02:12:39.708104 30122 raft_consensus.cc:348] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48156d6e37745f5835282c358bdddad" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46185 } }
I20260929 02:12:39.708447 30122 raft_consensus.cc:374] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:39.708551 30122 raft_consensus.cc:729] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48156d6e37745f5835282c358bdddad, State: Initialized, Role: FOLLOWER
I20260929 02:12:39.709044 30122 consensus_queue.cc:260] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48156d6e37745f5835282c358bdddad" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46185 } }
I20260929 02:12:39.709271 30122 raft_consensus.cc:388] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:39.709419 30122 raft_consensus.cc:482] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:39.709612 30122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:39.711741 30122 raft_consensus.cc:504] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48156d6e37745f5835282c358bdddad" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46185 } }
I20260929 02:12:39.712205 30122 leader_election.cc:302] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e48156d6e37745f5835282c358bdddad; no voters:
I20260929 02:12:39.712854 30122 leader_election.cc:288] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:39.712958 30127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:39.713724 30127 raft_consensus.cc:686] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [term 1 LEADER]: Becoming Leader. State: Replica: e48156d6e37745f5835282c358bdddad, State: Running, Role: LEADER
I20260929 02:12:39.714314 30127 consensus_queue.cc:237] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48156d6e37745f5835282c358bdddad" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46185 } }
I20260929 02:12:39.714838 30122 sys_catalog.cc:564] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:39.716905 30128 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e48156d6e37745f5835282c358bdddad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48156d6e37745f5835282c358bdddad" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46185 } } }
I20260929 02:12:39.717001 30129 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [sys.catalog]: SysCatalogTable state changed. Reason: New leader e48156d6e37745f5835282c358bdddad. Latest consensus state: current_term: 1 leader_uuid: "e48156d6e37745f5835282c358bdddad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48156d6e37745f5835282c358bdddad" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46185 } } }
I20260929 02:12:39.717269 30128 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:39.717272 30129 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:39.719251 30135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:39.724171 30135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:39.732846 30135 catalog_manager.cc:1347] Generated new cluster ID: c67e8e376a4b49c29779db27fc4c4a11
I20260929 02:12:39.732955 30135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:39.765170 30135 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:39.765573 30135 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:39.782560 30135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad: Generated new TSK 0
I20260929 02:12:39.783337 30135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:12:39.792078 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:46185
--builtin_ntp_servers=127.24.216.148:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 02:12:40.106240 30146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:40.106479 30146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:40.106621 30146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:40.113535 30146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:40.113624 30146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:40.113765 30146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:12:40.122999 30146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:46185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:40.124243 30146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:40.125970 30146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:40.138849 30152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:40.139113 30151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:40.139276 30154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:40.139750 30146 server_base.cc:1034] running on GCE node
I20260929 02:12:40.140347 30146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:40.141255 30146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:40.142480 30146 hybrid_clock.cc:648] HybridClock initialized: now 1790647960142433 us; error 62 us; skew 500 ppm
I20260929 02:12:40.142788 30146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:12:40.144943 30146 webserver.cc:460] Webserver started at http://127.24.216.129:36635/ using document root <none> and password file <none>
I20260929 02:12:40.145632 30146 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:40.145762 30146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:40.146097 30146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:40.148814 30146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "884d388fe44f4f298fec54440b7e8f12"
format_stamp: "Formatted at 2026-09-29 02:12:40 on dist-test-slave-9k0h"
server_key: "c55aefaab668fea092e4f64e4b984def"
server_key_iv: "9131375c66f7a617ef70403f0d4510a3"
server_key_version: "encryptionkey@0"
I20260929 02:12:40.149688 30146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "884d388fe44f4f298fec54440b7e8f12"
format_stamp: "Formatted at 2026-09-29 02:12:40 on dist-test-slave-9k0h"
server_key: "c55aefaab668fea092e4f64e4b984def"
server_key_iv: "9131375c66f7a617ef70403f0d4510a3"
server_key_version: "encryptionkey@0"
I20260929 02:12:40.154531 30146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 02:12:40.157613 30160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:40.158984 30146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:40.159173 30146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "884d388fe44f4f298fec54440b7e8f12"
format_stamp: "Formatted at 2026-09-29 02:12:40 on dist-test-slave-9k0h"
server_key: "c55aefaab668fea092e4f64e4b984def"
server_key_iv: "9131375c66f7a617ef70403f0d4510a3"
server_key_version: "encryptionkey@0"
I20260929 02:12:40.159439 30146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:40.164964 30146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:40.165750 30146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:40.165974 30146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:40.166958 30146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:40.168237 30146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:40.168326 30146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:40.168404 30146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:40.168457 30146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:40.199870 30146 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:38389
I20260929 02:12:40.199920 30272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:38389 every 8 connection(s)
I20260929 02:12:40.201177 30146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:12:40.208669 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:40.203593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:60889 (local address 127.24.216.190:46185)
0929 02:12:40.204024 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:40.204030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:40.205152 (+ 1122us) server_negotiation.cc:407] Connection header received
0929 02:12:40.206764 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:40.206768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:40.206822 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:40.206908 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:40.208121 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:40.208124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:40.208127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:40.208138 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:40.208170 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:40.208181 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:40.208265 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:40.208421 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 02:12:40.208466 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":213,"threads_started":1}
I20260929 02:12:40.209556 30276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:40.204177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:46185 (local address 127.24.216.129:60889)
0929 02:12:40.205012 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:40.205051 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:40.206526 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:40.207093 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:40.207101 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:40.207182 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:40.207851 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:12:40.207853 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:40.207889 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:12:40.207989 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:40.208261 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:40.208267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:40.208272 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:12:40.208465 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 02:12:40.208636 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":252,"threads_started":1}
I20260929 02:12:40.210378 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 30146
I20260929 02:12:40.210579 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:12:40.210984 30273 heartbeater.cc:347] Connected to a master server at 127.24.216.190:46185
I20260929 02:12:40.211248 30273 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:40.211364 25442 external_mini_cluster.cc:1428] Setting key ef70c5809c42d48ab8cedc6461b267c5
I20260929 02:12:40.211880 30273 heartbeater.cc:511] Master 127.24.216.190:46185 requested a full tablet report, sending...
I20260929 02:12:40.213869 30087 ts_manager.cc:194] Registered new tserver with Master: 884d388fe44f4f298fec54440b7e8f12 (127.24.216.129:38389)
I20260929 02:12:40.213946 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:46185
--builtin_ntp_servers=127.24.216.148:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 02:12:40.527755 30277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:40.528038 30277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:40.528123 30277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:40.535255 30277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:40.535362 30277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:40.535463 30277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:12:40.544111 30277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:46185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:40.545254 30277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:40.546947 30277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:40.559667 30285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:40.559608 30283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:40.559701 30282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:40.560635 30277 server_base.cc:1034] running on GCE node
I20260929 02:12:40.561134 30277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:40.562018 30277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:40.563235 30277 hybrid_clock.cc:648] HybridClock initialized: now 1790647960563191 us; error 56 us; skew 500 ppm
I20260929 02:12:40.563562 30277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:12:40.565614 30277 webserver.cc:460] Webserver started at http://127.24.216.130:45797/ using document root <none> and password file <none>
I20260929 02:12:40.566347 30277 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:40.566474 30277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:40.566797 30277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:40.569432 30277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "67797ae2d3fb467d99b92e8e7316c6da"
format_stamp: "Formatted at 2026-09-29 02:12:40 on dist-test-slave-9k0h"
server_key: "eb858e6a4ea33453b84876f3ef69d754"
server_key_iv: "fc2afd45005caff73e127fd23448e522"
server_key_version: "encryptionkey@0"
I20260929 02:12:40.570278 30277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "67797ae2d3fb467d99b92e8e7316c6da"
format_stamp: "Formatted at 2026-09-29 02:12:40 on dist-test-slave-9k0h"
server_key: "eb858e6a4ea33453b84876f3ef69d754"
server_key_iv: "fc2afd45005caff73e127fd23448e522"
server_key_version: "encryptionkey@0"
I20260929 02:12:40.575198 30277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:12:40.578456 30291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:40.579669 30277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:12:40.579844 30277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "67797ae2d3fb467d99b92e8e7316c6da"
format_stamp: "Formatted at 2026-09-29 02:12:40 on dist-test-slave-9k0h"
server_key: "eb858e6a4ea33453b84876f3ef69d754"
server_key_iv: "fc2afd45005caff73e127fd23448e522"
server_key_version: "encryptionkey@0"
I20260929 02:12:40.580041 30277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:40.585914 30277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:40.586668 30277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:40.586927 30277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:40.587740 30277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:40.589037 30277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:40.589146 30277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:40.589272 30277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:40.589378 30277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:40.620612 30277 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:33889
I20260929 02:12:40.620663 30403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:33889 every 8 connection(s)
I20260929 02:12:40.621920 30277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:12:40.622503 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 30277
I20260929 02:12:40.622663 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:12:40.623237 25442 external_mini_cluster.cc:1428] Setting key c1afa44064891e7992625cd9c543fd7e
I20260929 02:12:40.626255 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:46185
--builtin_ntp_servers=127.24.216.148:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 02:12:40.631237 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:40.624417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:34387 (local address 127.24.216.190:46185)
0929 02:12:40.624587 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:40.624593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:40.626312 (+ 1719us) server_negotiation.cc:407] Connection header received
0929 02:12:40.628967 (+ 2655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:40.628971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:40.629029 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:40.629107 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:40.630587 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:40.630591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:40.630594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:40.630608 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:40.630638 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:40.630648 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:40.630730 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:40.630973 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 02:12:40.631017 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:12:40.632395 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:40.625048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:46185 (local address 127.24.216.130:34387)
0929 02:12:40.626125 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:40.626167 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:40.628671 (+ 2504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:40.629376 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:40.629391 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:40.629507 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:40.630248 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:12:40.630254 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:40.630305 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:12:40.630416 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:40.630737 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:40.630749 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:40.630759 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:12:40.631068 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 02:12:40.631295 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":278,"threads_started":1}
I20260929 02:12:40.633694 30404 heartbeater.cc:347] Connected to a master server at 127.24.216.190:46185
I20260929 02:12:40.634029 30404 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:40.634629 30404 heartbeater.cc:511] Master 127.24.216.190:46185 requested a full tablet report, sending...
I20260929 02:12:40.635866 30087 ts_manager.cc:194] Registered new tserver with Master: 67797ae2d3fb467d99b92e8e7316c6da (127.24.216.130:33889)
W20260929 02:12:40.943281 30407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:40.943555 30407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:40.943639 30407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:40.950634 30407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:40.950728 30407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:40.950834 30407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:12:40.959692 30407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:46185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:40.960922 30407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:40.962719 30407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:40.975366 30415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:40.975462 30412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:40.975602 30413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:40.977000 30407 server_base.cc:1034] running on GCE node
I20260929 02:12:40.977579 30407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:40.978559 30407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:40.979782 30407 hybrid_clock.cc:648] HybridClock initialized: now 1790647960979743 us; error 52 us; skew 500 ppm
I20260929 02:12:40.980086 30407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:12:40.982275 30407 webserver.cc:460] Webserver started at http://127.24.216.131:37383/ using document root <none> and password file <none>
I20260929 02:12:40.982923 30407 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:40.983047 30407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:40.983422 30407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:40.986366 30407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "02a610179a844e89a712a532eef64514"
format_stamp: "Formatted at 2026-09-29 02:12:40 on dist-test-slave-9k0h"
server_key: "d0b6048440e1245cbc3217db37701df1"
server_key_iv: "3baae8bc4c508d0df9376823bd4e983e"
server_key_version: "encryptionkey@0"
I20260929 02:12:40.987174 30407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "02a610179a844e89a712a532eef64514"
format_stamp: "Formatted at 2026-09-29 02:12:40 on dist-test-slave-9k0h"
server_key: "d0b6048440e1245cbc3217db37701df1"
server_key_iv: "3baae8bc4c508d0df9376823bd4e983e"
server_key_version: "encryptionkey@0"
I20260929 02:12:40.991892 30407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 02:12:40.995165 30421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:40.996381 30407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:12:40.996562 30407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "02a610179a844e89a712a532eef64514"
format_stamp: "Formatted at 2026-09-29 02:12:40 on dist-test-slave-9k0h"
server_key: "d0b6048440e1245cbc3217db37701df1"
server_key_iv: "3baae8bc4c508d0df9376823bd4e983e"
server_key_version: "encryptionkey@0"
I20260929 02:12:40.996785 30407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:41.002282 30407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:41.003013 30407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:41.003286 30407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:41.004129 30407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:41.005489 30407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:41.005613 30407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:41.005745 30407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:41.005837 30407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:41.038403 30407 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:35883
I20260929 02:12:41.038443 30533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:35883 every 8 connection(s)
I20260929 02:12:41.039759 30407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:12:41.048631 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 30407
I20260929 02:12:41.048799 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:12:41.049237 25442 external_mini_cluster.cc:1428] Setting key fa9c2eae6acb0e7696183df11d5a37db
I20260929 02:12:41.052413 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.042216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:41235 (local address 127.24.216.190:46185)
0929 02:12:41.042414 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:41.042421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:41.043812 (+ 1391us) server_negotiation.cc:407] Connection header received
0929 02:12:41.045362 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:41.045366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:41.045413 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:41.045499 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:41.051576 (+ 6077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:41.051581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:41.051585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:41.051604 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:41.051642 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:41.051657 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:41.051798 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:41.052118 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 02:12:41.052165 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:12:41.053437 30536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.042807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:46185 (local address 127.24.216.131:41235)
0929 02:12:41.043670 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:41.043704 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:41.045136 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:41.045688 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:41.045695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:41.045780 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:41.051275 (+ 5495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:12:41.051278 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:41.051322 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:12:41.051410 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:41.051885 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:41.051895 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:41.051903 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 02:12:41.052205 (+ 302us) client_negotiation.cc:240] Negotiation successful
0929 02:12:41.052444 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":251,"threads_started":1}
I20260929 02:12:41.054728 30534 heartbeater.cc:347] Connected to a master server at 127.24.216.190:46185
I20260929 02:12:41.055015 30534 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:41.055580 30534 heartbeater.cc:511] Master 127.24.216.190:46185 requested a full tablet report, sending...
I20260929 02:12:41.056923 30087 ts_manager.cc:194] Registered new tserver with Master: 02a610179a844e89a712a532eef64514 (127.24.216.131:35883)
I20260929 02:12:41.064222 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:12:41.071242 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.068576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.24.216.190:46185)
0929 02:12:41.068739 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:41.068744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:41.069252 (+ 508us) server_negotiation.cc:407] Connection header received
0929 02:12:41.069473 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:41.069478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:41.069536 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:41.069652 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:41.070598 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:41.070604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:41.070609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:41.070626 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:41.070665 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:41.070676 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:41.070799 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:41.070921 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 02:12:41.070977 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:12:41.075054 30087 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:12:41.081672 30087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:12:41.100063 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.093106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:38389 (local address 127.0.0.1:60656)
0929 02:12:41.094931 (+ 1825us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:41.094991 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:41.095143 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:41.096373 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:41.096385 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:41.096496 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:41.097505 (+ 1009us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:12:41.097511 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:41.097547 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:12:41.097646 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:41.099079 (+ 1433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:41.099093 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:41.099105 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 02:12:41.099290 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 02:12:41.099406 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1508,"threads_started":1}
I20260929 02:12:41.101353 30547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.093304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.24.216.129:38389)
0929 02:12:41.095704 (+ 2400us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:41.095716 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:41.095751 (+ 35us) server_negotiation.cc:407] Connection header received
0929 02:12:41.095845 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:41.095855 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:41.096128 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:41.097061 (+ 933us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:41.098766 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:41.098778 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:41.098810 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:41.098838 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:41.098900 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:41.098926 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:41.100645 (+ 1719us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:41.100729 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 02:12:41.100925 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2207,"thread_start_us":2078,"threads_started":1}
I20260929 02:12:41.106321 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.096464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:33889 (local address 127.0.0.1:34676)
0929 02:12:41.102961 (+ 6497us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:41.102982 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:41.103100 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:41.103807 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:41.103815 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:41.103850 (+ 35us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:41.105201 (+ 1351us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:12:41.105205 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:41.105230 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:12:41.105270 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:41.105677 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:41.105682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:41.105688 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 02:12:41.105831 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 02:12:41.105878 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6307,"thread_start_us":6153,"threads_started":1}
I20260929 02:12:41.107151 30551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.097501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34676 (local address 127.24.216.130:33889)
0929 02:12:41.102454 (+ 4953us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:41.102466 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:41.103246 (+ 780us) server_negotiation.cc:407] Connection header received
0929 02:12:41.103343 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:41.103354 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:41.103614 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:41.103779 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:41.105427 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:41.105436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:41.105449 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:41.105467 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:41.105526 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:41.105553 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:41.106666 (+ 1113us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:41.106729 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 02:12:41.106842 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4747,"thread_start_us":4580,"threads_started":1}
I20260929 02:12:41.107940 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.093335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:35883 (local address 127.0.0.1:48760)
0929 02:12:41.100311 (+ 6976us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:41.100334 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:41.100429 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:41.105676 (+ 5247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:41.105681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:41.105702 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:41.106523 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:12:41.106526 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:41.106540 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:12:41.106580 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:41.107538 (+ 958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:41.107543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:41.107548 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:12:41.107638 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 02:12:41.107684 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6856,"thread_start_us":1115,"threads_started":1}
I20260929 02:12:41.108101 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.093829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.24.216.131:35883)
0929 02:12:41.105168 (+ 11339us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:41.105176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:41.105213 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:12:41.105276 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:41.105293 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:41.105509 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:41.105667 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:41.107319 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:41.107327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:41.107339 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:41.107357 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:41.107426 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:41.107446 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:41.107617 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:41.107675 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 02:12:41.107754 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10967,"thread_start_us":10802,"threads_started":1}
I20260929 02:12:41.109318 30208 tablet_service.cc:1467] Processing CreateTablet for tablet 515418494bec409181740b40ac0e5831 (DEFAULT_TABLE table=test-table [id=d39a52757cdb46e89c33b77301ee2af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:41.110590 30208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515418494bec409181740b40ac0e5831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:41.110695 30339 tablet_service.cc:1467] Processing CreateTablet for tablet 515418494bec409181740b40ac0e5831 (DEFAULT_TABLE table=test-table [id=d39a52757cdb46e89c33b77301ee2af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:41.111435 30469 tablet_service.cc:1467] Processing CreateTablet for tablet 515418494bec409181740b40ac0e5831 (DEFAULT_TABLE table=test-table [id=d39a52757cdb46e89c33b77301ee2af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:41.112056 30339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515418494bec409181740b40ac0e5831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:41.112738 30469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515418494bec409181740b40ac0e5831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:41.123359 30552 tablet_bootstrap.cc:492] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12: Bootstrap starting.
I20260929 02:12:41.126596 30552 tablet_bootstrap.cc:654] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:41.126894 30553 tablet_bootstrap.cc:492] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da: Bootstrap starting.
I20260929 02:12:41.127660 30554 tablet_bootstrap.cc:492] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514: Bootstrap starting.
I20260929 02:12:41.127944 30552 log.cc:824] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:41.130959 30553 tablet_bootstrap.cc:654] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:41.131229 30554 tablet_bootstrap.cc:654] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:41.131795 30552 tablet_bootstrap.cc:492] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12: No bootstrap required, opened a new log
I20260929 02:12:41.132062 30552 ts_tablet_manager.cc:1397] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260929 02:12:41.132071 30553 log.cc:824] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:41.132370 30554 log.cc:824] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:41.135407 30553 tablet_bootstrap.cc:492] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da: No bootstrap required, opened a new log
I20260929 02:12:41.135408 30554 tablet_bootstrap.cc:492] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514: No bootstrap required, opened a new log
I20260929 02:12:41.135670 30553 ts_tablet_manager.cc:1397] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 02:12:41.135670 30554 ts_tablet_manager.cc:1397] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260929 02:12:41.140173 30552 raft_consensus.cc:348] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884d388fe44f4f298fec54440b7e8f12" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38389 } } peers { permanent_uuid: "02a610179a844e89a712a532eef64514" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35883 } } peers { permanent_uuid: "67797ae2d3fb467d99b92e8e7316c6da" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33889 } }
I20260929 02:12:41.140551 30552 raft_consensus.cc:374] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:41.140681 30552 raft_consensus.cc:729] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884d388fe44f4f298fec54440b7e8f12, State: Initialized, Role: FOLLOWER
I20260929 02:12:41.141310 30552 consensus_queue.cc:260] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884d388fe44f4f298fec54440b7e8f12" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38389 } } peers { permanent_uuid: "02a610179a844e89a712a532eef64514" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35883 } } peers { permanent_uuid: "67797ae2d3fb467d99b92e8e7316c6da" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33889 } }
I20260929 02:12:41.142401 30273 heartbeater.cc:503] Master 127.24.216.190:46185 was elected leader, sending a full tablet report...
I20260929 02:12:41.143075 30552 ts_tablet_manager.cc:1428] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260929 02:12:41.146246 30554 raft_consensus.cc:348] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884d388fe44f4f298fec54440b7e8f12" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38389 } } peers { permanent_uuid: "02a610179a844e89a712a532eef64514" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35883 } } peers { permanent_uuid: "67797ae2d3fb467d99b92e8e7316c6da" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33889 } }
I20260929 02:12:41.146242 30553 raft_consensus.cc:348] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884d388fe44f4f298fec54440b7e8f12" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38389 } } peers { permanent_uuid: "02a610179a844e89a712a532eef64514" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35883 } } peers { permanent_uuid: "67797ae2d3fb467d99b92e8e7316c6da" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33889 } }
I20260929 02:12:41.146675 30553 raft_consensus.cc:374] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:41.146675 30554 raft_consensus.cc:374] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:41.146780 30554 raft_consensus.cc:729] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a610179a844e89a712a532eef64514, State: Initialized, Role: FOLLOWER
I20260929 02:12:41.146795 30553 raft_consensus.cc:729] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67797ae2d3fb467d99b92e8e7316c6da, State: Initialized, Role: FOLLOWER
I20260929 02:12:41.147392 30553 consensus_queue.cc:260] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884d388fe44f4f298fec54440b7e8f12" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38389 } } peers { permanent_uuid: "02a610179a844e89a712a532eef64514" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35883 } } peers { permanent_uuid: "67797ae2d3fb467d99b92e8e7316c6da" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33889 } }
I20260929 02:12:41.147392 30554 consensus_queue.cc:260] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884d388fe44f4f298fec54440b7e8f12" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38389 } } peers { permanent_uuid: "02a610179a844e89a712a532eef64514" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35883 } } peers { permanent_uuid: "67797ae2d3fb467d99b92e8e7316c6da" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33889 } }
I20260929 02:12:41.149459 30404 heartbeater.cc:503] Master 127.24.216.190:46185 was elected leader, sending a full tablet report...
I20260929 02:12:41.149570 30534 heartbeater.cc:503] Master 127.24.216.190:46185 was elected leader, sending a full tablet report...
I20260929 02:12:41.149603 30554 ts_tablet_manager.cc:1428] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 02:12:41.150009 30553 ts_tablet_manager.cc:1428] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260929 02:12:41.203506 30274 tablet.cc:2369] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:12:41.291303 30535 tablet.cc:2369] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:12:41.336298 30559 raft_consensus.cc:482] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:12:41.336578 30559 raft_consensus.cc:504] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884d388fe44f4f298fec54440b7e8f12" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38389 } } peers { permanent_uuid: "02a610179a844e89a712a532eef64514" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35883 } } peers { permanent_uuid: "67797ae2d3fb467d99b92e8e7316c6da" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33889 } }
I20260929 02:12:41.338359 30559 leader_election.cc:288] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 884d388fe44f4f298fec54440b7e8f12 (127.24.216.129:38389), 67797ae2d3fb467d99b92e8e7316c6da (127.24.216.130:33889)
I20260929 02:12:41.341001 30536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.338638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:33889 (local address 127.24.216.131:48233)
0929 02:12:41.338784 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:41.338805 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:41.338905 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:41.339607 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:41.339612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:41.339636 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:41.340275 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:12:41.340278 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:41.340297 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:12:41.340331 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:41.340600 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:41.340603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:41.340606 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 02:12:41.340716 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 02:12:41.340768 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 02:12:41.341035 30551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.339093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:48233 (local address 127.24.216.130:33889)
0929 02:12:41.339278 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:41.339292 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:41.339311 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:12:41.339382 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:41.339387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:41.339454 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:41.339553 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:41.340463 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:41.340467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:41.340470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:41.340485 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:41.340510 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:41.340521 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:41.340615 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:41.340734 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 02:12:41.340789 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:12:41.341593 30547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.338691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:34625 (local address 127.24.216.129:38389)
0929 02:12:41.338885 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:41.338893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:41.339493 (+ 600us) server_negotiation.cc:407] Connection header received
0929 02:12:41.339613 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:41.339617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:41.339674 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:41.339782 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:41.340895 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:41.340900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:41.340905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:41.340922 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:41.340960 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:41.340971 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:41.341086 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:41.341259 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 02:12:41.341318 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:12:41.341722 30561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.338684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:38389 (local address 127.24.216.131:34625)
0929 02:12:41.339389 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:41.339405 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:41.339507 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:41.339816 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:41.339819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:41.339835 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:41.340713 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:12:41.340715 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:41.340726 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:12:41.340756 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:41.341062 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:41.341066 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:41.341070 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 02:12:41.341240 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 02:12:41.341292 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":359,"threads_started":1}
I20260929 02:12:41.341992 30359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515418494bec409181740b40ac0e5831" candidate_uuid: "02a610179a844e89a712a532eef64514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67797ae2d3fb467d99b92e8e7316c6da" is_pre_election: true
I20260929 02:12:41.342588 30359 raft_consensus.cc:2454] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02a610179a844e89a712a532eef64514 in term 0.
I20260929 02:12:41.342710 30228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515418494bec409181740b40ac0e5831" candidate_uuid: "02a610179a844e89a712a532eef64514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884d388fe44f4f298fec54440b7e8f12" is_pre_election: true
I20260929 02:12:41.343222 30228 raft_consensus.cc:2454] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02a610179a844e89a712a532eef64514 in term 0.
I20260929 02:12:41.343223 30425 leader_election.cc:302] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a610179a844e89a712a532eef64514, 67797ae2d3fb467d99b92e8e7316c6da; no voters:
I20260929 02:12:41.343580 30559 raft_consensus.cc:2781] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:12:41.343735 30559 raft_consensus.cc:482] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:12:41.343861 30559 raft_consensus.cc:3037] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:41.345995 30559 raft_consensus.cc:504] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884d388fe44f4f298fec54440b7e8f12" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38389 } } peers { permanent_uuid: "02a610179a844e89a712a532eef64514" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35883 } } peers { permanent_uuid: "67797ae2d3fb467d99b92e8e7316c6da" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33889 } }
I20260929 02:12:41.346578 30559 leader_election.cc:288] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [CANDIDATE]: Term 1 election: Requested vote from peers 884d388fe44f4f298fec54440b7e8f12 (127.24.216.129:38389), 67797ae2d3fb467d99b92e8e7316c6da (127.24.216.130:33889)
I20260929 02:12:41.346905 30228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515418494bec409181740b40ac0e5831" candidate_uuid: "02a610179a844e89a712a532eef64514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884d388fe44f4f298fec54440b7e8f12"
I20260929 02:12:41.347025 30359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515418494bec409181740b40ac0e5831" candidate_uuid: "02a610179a844e89a712a532eef64514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67797ae2d3fb467d99b92e8e7316c6da"
I20260929 02:12:41.347100 30228 raft_consensus.cc:3037] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:41.347234 30359 raft_consensus.cc:3037] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:41.349200 30228 raft_consensus.cc:2454] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02a610179a844e89a712a532eef64514 in term 1.
I20260929 02:12:41.349417 30359 raft_consensus.cc:2454] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02a610179a844e89a712a532eef64514 in term 1.
I20260929 02:12:41.349581 30422 leader_election.cc:302] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a610179a844e89a712a532eef64514, 884d388fe44f4f298fec54440b7e8f12; no voters:
I20260929 02:12:41.349854 30559 raft_consensus.cc:2781] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:41.350315 30559 raft_consensus.cc:686] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [term 1 LEADER]: Becoming Leader. State: Replica: 02a610179a844e89a712a532eef64514, State: Running, Role: LEADER
I20260929 02:12:41.350818 30559 consensus_queue.cc:237] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884d388fe44f4f298fec54440b7e8f12" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38389 } } peers { permanent_uuid: "02a610179a844e89a712a532eef64514" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35883 } } peers { permanent_uuid: "67797ae2d3fb467d99b92e8e7316c6da" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33889 } }
I20260929 02:12:41.355115 30087 catalog_manager.cc:5495] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02a610179a844e89a712a532eef64514 (127.24.216.131). New cstate: current_term: 1 leader_uuid: "02a610179a844e89a712a532eef64514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884d388fe44f4f298fec54440b7e8f12" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02a610179a844e89a712a532eef64514" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67797ae2d3fb467d99b92e8e7316c6da" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33889 } health_report { overall_health: UNKNOWN } } }
W20260929 02:12:41.374001 30405 tablet.cc:2369] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:12:41.401041 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.398620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.24.216.131:35883)
0929 02:12:41.398821 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:41.398828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:41.398939 (+ 111us) server_negotiation.cc:407] Connection header received
0929 02:12:41.399217 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:41.399222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:41.399316 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:41.399455 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:41.400369 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:41.400375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:41.400381 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:41.400404 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:41.400444 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:41.400460 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:41.400601 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:41.400689 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 02:12:41.400750 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:12:41.409194 30228 raft_consensus.cc:1261] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12 [term 1 FOLLOWER]: Refusing update from remote peer 02a610179a844e89a712a532eef64514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:41.409193 30359 raft_consensus.cc:1261] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da [term 1 FOLLOWER]: Refusing update from remote peer 02a610179a844e89a712a532eef64514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:41.410121 30562 consensus_queue.cc:1035] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "884d388fe44f4f298fec54440b7e8f12" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:41.410493 30559 consensus_queue.cc:1035] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67797ae2d3fb467d99b92e8e7316c6da" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:41.423164 30567 mvcc.cc:205] Tried to move back new op lower bound from 7334494049922105344 to 7334494049696817152. Current Snapshot: MvccSnapshot[applied={T|T < 7334494049922105344}]
I20260929 02:12:41.423996 30565 mvcc.cc:205] Tried to move back new op lower bound from 7334494049922105344 to 7334494049696817152. Current Snapshot: MvccSnapshot[applied={T|T < 7334494049922105344}]
I20260929 02:12:41.428584 30566 mvcc.cc:205] Tried to move back new op lower bound from 7334494049922105344 to 7334494049696817152. Current Snapshot: MvccSnapshot[applied={T|T < 7334494049922105344}]
I20260929 02:12:41.438766 30086 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:41.439060 30086 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:41.442822 30086 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e48156d6e37745f5835282c358bdddad: Sending DeleteTablet for 3 replicas of tablet 515418494bec409181740b40ac0e5831
I20260929 02:12:41.443862 30208 tablet_service.cc:1514] Processing DeleteTablet for tablet 515418494bec409181740b40ac0e5831 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:41 UTC) from {username='slave'} at 127.0.0.1:60656
I20260929 02:12:41.443876 30469 tablet_service.cc:1514] Processing DeleteTablet for tablet 515418494bec409181740b40ac0e5831 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:41 UTC) from {username='slave'} at 127.0.0.1:48760
I20260929 02:12:41.444208 30339 tablet_service.cc:1514] Processing DeleteTablet for tablet 515418494bec409181740b40ac0e5831 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:41 UTC) from {username='slave'} at 127.0.0.1:34676
I20260929 02:12:41.446116 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 30146
I20260929 02:12:41.456041 30575 tablet_replica.cc:330] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12: stopping tablet replica
I20260929 02:12:41.456477 30575 raft_consensus.cc:2229] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:41.456971 30575 raft_consensus.cc:2258] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:41.457728 30547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.447839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.24.216.129:38389)
0929 02:12:41.448079 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:41.448087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:41.454933 (+ 6846us) server_negotiation.cc:407] Connection header received
0929 02:12:41.455016 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:41.455024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:41.455156 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:41.455283 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:41.456597 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:41.456606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:41.456614 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:41.456640 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:41.456691 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:41.456712 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:41.456842 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:41.457298 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 02:12:41.457375 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:12:41.457741 30577 tablet_replica.cc:330] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da: stopping tablet replica
I20260929 02:12:41.458122 30577 raft_consensus.cc:2229] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:41.459333 30268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:41.460831 30577 raft_consensus.cc:2258] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:41.462844 30577 ts_tablet_manager.cc:1905] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:41.464321 30576 tablet_replica.cc:330] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514: stopping tablet replica
I20260929 02:12:41.464677 30576 raft_consensus.cc:2229] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:12:41.465310 30576 raft_consensus.cc:2258] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:41.474404 30577 ts_tablet_manager.cc:1918] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:12:41.474471 30576 ts_tablet_manager.cc:1905] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:41.474572 30577 log.cc:1196] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/515418494bec409181740b40ac0e5831
I20260929 02:12:41.475102 30577 ts_tablet_manager.cc:1939] T 515418494bec409181740b40ac0e5831 P 67797ae2d3fb467d99b92e8e7316c6da: Deleting consensus metadata
I20260929 02:12:41.476913 30074 catalog_manager.cc:4841] TS 67797ae2d3fb467d99b92e8e7316c6da (127.24.216.130:33889): tablet 515418494bec409181740b40ac0e5831 (table test-table [id=d39a52757cdb46e89c33b77301ee2af2]) successfully deleted
I20260929 02:12:41.482726 30576 ts_tablet_manager.cc:1918] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:12:41.482897 30576 log.cc:1196] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/515418494bec409181740b40ac0e5831
I20260929 02:12:41.483390 30576 ts_tablet_manager.cc:1939] T 515418494bec409181740b40ac0e5831 P 02a610179a844e89a712a532eef64514: Deleting consensus metadata
I20260929 02:12:41.486831 30073 catalog_manager.cc:4841] TS 02a610179a844e89a712a532eef64514 (127.24.216.131:35883): tablet 515418494bec409181740b40ac0e5831 (table test-table [id=d39a52757cdb46e89c33b77301ee2af2]) successfully deleted
I20260929 02:12:41.460608 30575 ts_tablet_manager.cc:1905] T 515418494bec409181740b40ac0e5831 P 884d388fe44f4f298fec54440b7e8f12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:41.556913 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 30146
W20260929 02:12:41.574446 30071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.216.129:38389 (error 108)
I20260929 02:12:41.574563 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 30277
I20260929 02:12:41.576269 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.575760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:38389 (local address 127.0.0.1:60678)
0929 02:12:41.576022 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:41.576125 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:38389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260929 02:12:41.576519 30071 catalog_manager.cc:4584] TS 884d388fe44f4f298fec54440b7e8f12 (127.24.216.129:38389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 515418494bec409181740b40ac0e5831: Network error: Client connection negotiation failed: client connection to 127.24.216.129:38389: connect: Connection refused (error 111)
I20260929 02:12:41.578040 30551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.575895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.24.216.130:33889)
0929 02:12:41.576093 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:41.576100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:41.576121 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:12:41.576185 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:41.576190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:41.576295 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:41.576399 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:41.577434 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:41.577440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:41.577445 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:41.577464 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:41.577502 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:41.577517 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:41.577647 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:41.577702 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 02:12:41.577757 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:12:41.578820 30399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:41.630198 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.629611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:38389 (local address 127.0.0.1:60684)
0929 02:12:41.629869 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:41.629982 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:38389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 02:12:41.631335 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.630954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:38389 (local address 127.0.0.1:60700)
0929 02:12:41.631109 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:41.631197 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:38389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 02:12:41.666311 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 30277
I20260929 02:12:41.690256 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 30407
I20260929 02:12:41.692104 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.691565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:38389 (local address 127.0.0.1:60706)
0929 02:12:41.691788 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:41.691915 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:38389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260929 02:12:41.693334 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.692901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:38389 (local address 127.0.0.1:60714)
0929 02:12:41.693095 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:41.693180 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:38389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 02:12:41.693842 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.691068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.24.216.131:35883)
0929 02:12:41.691256 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:41.691262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:41.691653 (+ 391us) server_negotiation.cc:407] Connection header received
0929 02:12:41.691740 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:41.691745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:41.691811 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:41.691903 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:41.692963 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:41.692969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:41.692973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:41.692990 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:41.693024 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:41.693040 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:41.693167 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:41.693507 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 02:12:41.693559 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:12:41.694692 30529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:41.774075 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.773522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:38389 (local address 127.0.0.1:60728)
0929 02:12:41.773760 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:41.773879 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:38389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 02:12:41.775223 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:41.774765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:38389 (local address 127.0.0.1:60744)
0929 02:12:41.774973 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:41.775085 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:38389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 02:12:41.809187 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 30407
I20260929 02:12:41.828544 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 30056
I20260929 02:12:41.829679 30117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:41.932420 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 30056
2026-09-29T02:12:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2705 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 02:12:41.973426 25442 mini_oidc.cc:152] Starting JWKS server
I20260929 02:12:41.976025 25442 webserver.cc:460] Webserver started at https://127.0.0.1:43589/ using document root <none> and password file <none>
I20260929 02:12:41.976365 25442 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:12:41.976989 25442 webserver.cc:460] Webserver started at http://127.0.0.1:44527/ using document root <none> and password file <none>
2026-09-29T02:12:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:41Z Disabled control of system clock
I20260929 02:12:42.002921 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:42133
--webserver_interface=127.24.216.190
--webserver_port=0
--builtin_ntp_servers=127.24.216.148:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:42133
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43589/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/testchainca.cert with env {}
W20260929 02:12:42.314939 30589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:42.315207 30589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:42.315377 30589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:42.322397 30589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:42.322513 30589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:42.322577 30589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:42.322677 30589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:42.322737 30589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:12:42.322851 30589 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:12:42.322908 30589 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43589/jwks.json
I20260929 02:12:42.332379 30589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:42133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:42133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43589/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:42.333613 30589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:42.334959 30589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:42.346465 30594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:42.346529 30595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:42.346551 30597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:42.347378 30589 server_base.cc:1034] running on GCE node
I20260929 02:12:42.348081 30589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:42.349431 30589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:42.350647 30589 hybrid_clock.cc:648] HybridClock initialized: now 1790647962350611 us; error 50 us; skew 500 ppm
I20260929 02:12:42.350978 30589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:12:42.353137 30589 webserver.cc:460] Webserver started at http://127.24.216.190:43527/ using document root <none> and password file <none>
I20260929 02:12:42.353788 30589 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:42.353920 30589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:42.354287 30589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:42.357023 30589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "82f33382bb3b4fe2a1bf1253816e6f62"
format_stamp: "Formatted at 2026-09-29 02:12:42 on dist-test-slave-9k0h"
server_key: "8d75ae75f667b51c13f92c2756ab2b69"
server_key_iv: "6e3970c3310cc80b50d2e65d2fb4b403"
server_key_version: "encryptionkey@0"
I20260929 02:12:42.357786 30589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "82f33382bb3b4fe2a1bf1253816e6f62"
format_stamp: "Formatted at 2026-09-29 02:12:42 on dist-test-slave-9k0h"
server_key: "8d75ae75f667b51c13f92c2756ab2b69"
server_key_iv: "6e3970c3310cc80b50d2e65d2fb4b403"
server_key_version: "encryptionkey@0"
I20260929 02:12:42.362692 30589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:12:42.365927 30603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:42.367167 30589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 02:12:42.367357 30589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "82f33382bb3b4fe2a1bf1253816e6f62"
format_stamp: "Formatted at 2026-09-29 02:12:42 on dist-test-slave-9k0h"
server_key: "8d75ae75f667b51c13f92c2756ab2b69"
server_key_iv: "6e3970c3310cc80b50d2e65d2fb4b403"
server_key_version: "encryptionkey@0"
I20260929 02:12:42.367558 30589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:42.401641 30589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:42.402565 30589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:42.402874 30589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:42.420800 30589 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:42133
I20260929 02:12:42.420802 30654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:42133 every 8 connection(s)
I20260929 02:12:42.422627 30589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:42.426824 30655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:42.430277 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 30589
I20260929 02:12:42.430493 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:42.431085 25442 external_mini_cluster.cc:1428] Setting key a75f845fdc4d9f3639d3060d7c810143
I20260929 02:12:42.437572 30655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62: Bootstrap starting.
I20260929 02:12:42.441084 30655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:42.442301 30655 log.cc:824] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:42.445353 30655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62: No bootstrap required, opened a new log
I20260929 02:12:42.445597 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:42.432882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.24.216.190:42133)
0929 02:12:42.433884 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:42.433895 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:42.433933 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:12:42.436161 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:42.436183 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:42.436598 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:42.436905 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:42.437522 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:42.438921 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:42.440807 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:42.441529 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:42.441659 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:42.443093 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:42.443099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:42.443112 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:42.443713 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:42.443822 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:42.443846 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:42.443957 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:42.444137 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 02:12:42.444430 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":359,"threads_started":1}
I20260929 02:12:42.449165 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:12:42.453330 30655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f33382bb3b4fe2a1bf1253816e6f62" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42133 } }
I20260929 02:12:42.453678 30655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:42.453778 30655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f33382bb3b4fe2a1bf1253816e6f62, State: Initialized, Role: FOLLOWER
I20260929 02:12:42.454351 30655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f33382bb3b4fe2a1bf1253816e6f62" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42133 } }
I20260929 02:12:42.454583 30655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:42.454722 30655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:42.454890 30655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:42.456976 30655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f33382bb3b4fe2a1bf1253816e6f62" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42133 } }
I20260929 02:12:42.457442 30655 leader_election.cc:302] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82f33382bb3b4fe2a1bf1253816e6f62; no voters:
I20260929 02:12:42.458043 30655 leader_election.cc:288] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:42.458212 30660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:42.458972 30660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [term 1 LEADER]: Becoming Leader. State: Replica: 82f33382bb3b4fe2a1bf1253816e6f62, State: Running, Role: LEADER
I20260929 02:12:42.459548 30660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f33382bb3b4fe2a1bf1253816e6f62" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42133 } }
I20260929 02:12:42.460053 30655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:42.462234 30661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82f33382bb3b4fe2a1bf1253816e6f62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f33382bb3b4fe2a1bf1253816e6f62" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42133 } } }
I20260929 02:12:42.462616 30661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:42.462406 30662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82f33382bb3b4fe2a1bf1253816e6f62. Latest consensus state: current_term: 1 leader_uuid: "82f33382bb3b4fe2a1bf1253816e6f62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f33382bb3b4fe2a1bf1253816e6f62" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42133 } } }
I20260929 02:12:42.466269 30662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:42.467155 30665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:42.472532 30665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:42.481037 30665 catalog_manager.cc:1347] Generated new cluster ID: 57e7d192b7a14b59804bb79def5fa3ff
I20260929 02:12:42.481154 30665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:42.492339 30665 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:42.493510 30665 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:42.503585 30665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82f33382bb3b4fe2a1bf1253816e6f62: Generated new TSK 0
I20260929 02:12:42.504307 30665 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 02:12:42.537724 30685 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 02:12:42.537904 30685 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 02:12:42.539079 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:42.517955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.24.216.190:42133)
0929 02:12:42.518137 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:42.518162 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:42.518589 (+ 427us) server_negotiation.cc:407] Connection header received
0929 02:12:42.518874 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:42.518883 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:42.518970 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:42.519122 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:12:42.519633 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:42.520737 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:42.522172 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:42.522831 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:42.522967 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:42.523407 (+ 440us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:12:42.538453 (+ 15046us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 02:12:42.538764 (+ 311us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38396: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 02:12:42.539284 30658 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38396: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 02:12:42.540732 30684 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:12:42.517820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:42133 (local address 127.0.0.1:38396)
0929 02:12:42.518492 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:42.518508 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:42.518644 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:42.519150 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:42.519154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:42.519162 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:12:42.519494 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:42.519502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:42.520878 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:42.520881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:42.522012 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:42.522022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:42.523002 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:42.523007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:42.523116 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:42.523169 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:12:42.538824 (+ 15655us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 02:12:42.539309 (+ 485us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.216.190:42133: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":375,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.24.216.190:42133: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43589/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 02:12:42.543216 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 30589
I20260929 02:12:42.546895 30650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:42.668838 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 30589
2026-09-29T02:12:42Z chronyd exiting
I20260929 02:12:42.719916 25442 test_util.cc:175] -----------------------------------------------
I20260929 02:12:42.720045 25442 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790647920678408-25442-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (750 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 02:12:42.723598 25442 mini_oidc.cc:152] Starting JWKS server
I20260929 02:12:42.725343 25442 webserver.cc:460] Webserver started at https://127.0.0.1:44093/ using document root <none> and password file <none>
I20260929 02:12:42.725517 25442 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:12:42.726063 25442 webserver.cc:460] Webserver started at http://127.0.0.1:34577/ using document root <none> and password file <none>
2026-09-29T02:12:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:42Z Disabled control of system clock
I20260929 02:12:42.752362 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:46131
--webserver_interface=127.24.216.190
--webserver_port=0
--builtin_ntp_servers=127.24.216.148:34919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:46131
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44093/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/testchainca.cert with env {}
W20260929 02:12:43.062064 30692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:43.062402 30692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:43.062487 30692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:43.069558 30692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:43.069653 30692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:43.069689 30692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:43.069744 30692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:43.069780 30692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:12:43.069849 30692 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:12:43.069888 30692 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44093/jwks.json
I20260929 02:12:43.079540 30692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:34919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:46131
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:46131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44093/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:43.080658 30692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:43.081851 30692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:43.093000 30698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:43.093046 30697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:43.093067 30700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:43.093919 30692 server_base.cc:1034] running on GCE node
I20260929 02:12:43.094645 30692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:43.095873 30692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:43.097066 30692 hybrid_clock.cc:648] HybridClock initialized: now 1790647963097032 us; error 34 us; skew 500 ppm
I20260929 02:12:43.097352 30692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:12:43.099416 30692 webserver.cc:460] Webserver started at http://127.24.216.190:43377/ using document root <none> and password file <none>
I20260929 02:12:43.100029 30692 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:43.100114 30692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:43.100364 30692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:43.103099 30692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "07bee6fdd8a64a7fb9e5c5727c0e81f0"
format_stamp: "Formatted at 2026-09-29 02:12:43 on dist-test-slave-9k0h"
server_key: "98e60f1ce772f177c27a12a606c61356"
server_key_iv: "bd36d22bf184f20d63354b42250a061d"
server_key_version: "encryptionkey@0"
I20260929 02:12:43.103780 30692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "07bee6fdd8a64a7fb9e5c5727c0e81f0"
format_stamp: "Formatted at 2026-09-29 02:12:43 on dist-test-slave-9k0h"
server_key: "98e60f1ce772f177c27a12a606c61356"
server_key_iv: "bd36d22bf184f20d63354b42250a061d"
server_key_version: "encryptionkey@0"
I20260929 02:12:43.108345 30692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:12:43.111456 30706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:43.112789 30692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:12:43.113029 30692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "07bee6fdd8a64a7fb9e5c5727c0e81f0"
format_stamp: "Formatted at 2026-09-29 02:12:43 on dist-test-slave-9k0h"
server_key: "98e60f1ce772f177c27a12a606c61356"
server_key_iv: "bd36d22bf184f20d63354b42250a061d"
server_key_version: "encryptionkey@0"
I20260929 02:12:43.113243 30692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:43.134063 30692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:43.134886 30692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:43.135195 30692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:43.152733 30692 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:46131
I20260929 02:12:43.152757 30757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:46131 every 8 connection(s)
I20260929 02:12:43.154007 30692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:43.157625 30758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:43.160089 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 30692
I20260929 02:12:43.160279 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:43.161088 25442 external_mini_cluster.cc:1428] Setting key b2cc2536cd58db5de850388c2cec397c
I20260929 02:12:43.169703 30758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0: Bootstrap starting.
I20260929 02:12:43.173260 30758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:43.174384 30758 log.cc:824] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:43.174559 30761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:43.163013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.24.216.190:46131)
0929 02:12:43.164027 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:43.164036 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:43.164075 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:12:43.165686 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:43.165708 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:43.166205 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:43.166539 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:43.167097 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:43.168382 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:43.170110 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:43.170814 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:43.170945 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:43.172265 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:43.172272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:43.172285 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:43.172796 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:43.172881 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:43.172901 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:43.172986 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:43.173230 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 02:12:43.173560 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":407,"threads_started":1}
I20260929 02:12:43.177179 30758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0: No bootstrap required, opened a new log
I20260929 02:12:43.178061 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:12:43.184475 30758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bee6fdd8a64a7fb9e5c5727c0e81f0" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46131 } }
I20260929 02:12:43.184924 30758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:43.185037 30758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bee6fdd8a64a7fb9e5c5727c0e81f0, State: Initialized, Role: FOLLOWER
I20260929 02:12:43.185621 30758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bee6fdd8a64a7fb9e5c5727c0e81f0" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46131 } }
I20260929 02:12:43.185863 30758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:43.185999 30758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:43.186214 30758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:43.188308 30758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bee6fdd8a64a7fb9e5c5727c0e81f0" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46131 } }
I20260929 02:12:43.188879 30758 leader_election.cc:302] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07bee6fdd8a64a7fb9e5c5727c0e81f0; no voters:
I20260929 02:12:43.189497 30758 leader_election.cc:288] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:43.189563 30763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:43.190328 30763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [term 1 LEADER]: Becoming Leader. State: Replica: 07bee6fdd8a64a7fb9e5c5727c0e81f0, State: Running, Role: LEADER
I20260929 02:12:43.190804 30763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bee6fdd8a64a7fb9e5c5727c0e81f0" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46131 } }
I20260929 02:12:43.191435 30758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:43.193389 30764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07bee6fdd8a64a7fb9e5c5727c0e81f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bee6fdd8a64a7fb9e5c5727c0e81f0" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46131 } } }
I20260929 02:12:43.193471 30765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07bee6fdd8a64a7fb9e5c5727c0e81f0. Latest consensus state: current_term: 1 leader_uuid: "07bee6fdd8a64a7fb9e5c5727c0e81f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bee6fdd8a64a7fb9e5c5727c0e81f0" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46131 } } }
I20260929 02:12:43.194613 30765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:43.194650 30764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:43.199098 30770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:43.203140 30770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:43.210686 30770 catalog_manager.cc:1347] Generated new cluster ID: 6ed3f2fa998244f2a29a56ebdd2bc9c5
I20260929 02:12:43.210781 30770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:43.223850 30770 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:43.224936 30770 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:43.231307 30770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07bee6fdd8a64a7fb9e5c5727c0e81f0: Generated new TSK 0
I20260929 02:12:43.232070 30770 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 02:12:43.274039 30788 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 02:12:43.274191 30788 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260929 02:12:43.275388 30787 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:12:43.255112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:46131 (local address 127.0.0.1:60202)
0929 02:12:43.255721 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:43.255736 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:43.255856 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:43.256285 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:43.256290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:43.256299 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:12:43.256522 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:43.256529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:43.257946 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:43.257949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:43.259081 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:43.259092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:43.260099 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:43.260103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:43.260239 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:43.260268 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:12:43.274934 (+ 14666us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44093/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 02:12:43.275110 (+ 176us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.216.190:46131: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44093/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":357,"threads_started":1}
I20260929 02:12:43.275386 30761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:43.255272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60202 (local address 127.24.216.190:46131)
0929 02:12:43.255461 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:43.255467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:43.255822 (+ 355us) server_negotiation.cc:407] Connection header received
0929 02:12:43.256029 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:43.256035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:43.256122 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:43.256284 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:12:43.256690 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:43.257815 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:43.259216 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:43.259945 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:43.260064 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:43.260500 (+ 436us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:12:43.274719 (+ 14219us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44093/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 02:12:43.275096 (+ 377us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60202: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44093/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 02:12:43.275530 30761 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60202: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44093/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 02:12:43.277048 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 30692
I20260929 02:12:43.278038 30753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:43.367731 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 30692
2026-09-29T02:12:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (698 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 02:12:43.422731 25442 mini_oidc.cc:152] Starting JWKS server
I20260929 02:12:43.424510 25442 webserver.cc:460] Webserver started at https://127.0.0.1:33857/ using document root <none> and password file <none>
I20260929 02:12:43.424801 25442 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:12:43.425631 25442 webserver.cc:460] Webserver started at http://127.0.0.1:44711/ using document root <none> and password file <none>
2026-09-29T02:12:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:43Z Disabled control of system clock
I20260929 02:12:43.452521 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:42561
--webserver_interface=127.24.216.190
--webserver_port=0
--builtin_ntp_servers=127.24.216.148:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:42561
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33857/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 02:12:43.762588 30795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:43.762882 30795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:43.762971 30795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:43.769915 30795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:43.770035 30795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:43.770078 30795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:43.770134 30795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:43.770248 30795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:12:43.770325 30795 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:12:43.770368 30795 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33857/jwks.json
I20260929 02:12:43.779460 30795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:42561
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:42561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33857/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:43.780717 30795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:43.782490 30795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:43.794395 30800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:43.794348 30801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:43.794492 30803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:43.795054 30795 server_base.cc:1034] running on GCE node
I20260929 02:12:43.795831 30795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:43.797184 30795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:43.798403 30795 hybrid_clock.cc:648] HybridClock initialized: now 1790647963798373 us; error 60 us; skew 500 ppm
I20260929 02:12:43.798730 30795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:12:43.805145 30795 webserver.cc:460] Webserver started at http://127.24.216.190:33557/ using document root <none> and password file <none>
I20260929 02:12:43.805898 30795 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:43.806025 30795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:43.806411 30795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:43.809218 30795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "bc28eb514483451c81fc1f2106ef89a9"
format_stamp: "Formatted at 2026-09-29 02:12:43 on dist-test-slave-9k0h"
server_key: "cb97005497a871fd1bd644ff73d0ae77"
server_key_iv: "a8d922204e1c0f8458c5edb8ca0a7cc0"
server_key_version: "encryptionkey@0"
I20260929 02:12:43.809994 30795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "bc28eb514483451c81fc1f2106ef89a9"
format_stamp: "Formatted at 2026-09-29 02:12:43 on dist-test-slave-9k0h"
server_key: "cb97005497a871fd1bd644ff73d0ae77"
server_key_iv: "a8d922204e1c0f8458c5edb8ca0a7cc0"
server_key_version: "encryptionkey@0"
I20260929 02:12:43.814968 30795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 02:12:43.818107 30809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:43.819419 30795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:12:43.819599 30795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "bc28eb514483451c81fc1f2106ef89a9"
format_stamp: "Formatted at 2026-09-29 02:12:43 on dist-test-slave-9k0h"
server_key: "cb97005497a871fd1bd644ff73d0ae77"
server_key_iv: "a8d922204e1c0f8458c5edb8ca0a7cc0"
server_key_version: "encryptionkey@0"
I20260929 02:12:43.819815 30795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:43.836712 30795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:43.837493 30795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:43.837872 30795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:43.855406 30795 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:42561
I20260929 02:12:43.855413 30860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:42561 every 8 connection(s)
I20260929 02:12:43.856673 30795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:43.860040 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 30795
I20260929 02:12:43.860241 30861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:43.860250 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:43.860888 25442 external_mini_cluster.cc:1428] Setting key e1bd2a7ebd825bd731fc6ed559fa845d
I20260929 02:12:43.874105 30861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9: Bootstrap starting.
I20260929 02:12:43.876369 30864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:43.862695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.24.216.190:42561)
0929 02:12:43.863778 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:43.863806 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:43.863858 (+ 52us) server_negotiation.cc:407] Connection header received
0929 02:12:43.866523 (+ 2665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:43.866549 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:43.867123 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:43.867574 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:43.868051 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:43.869932 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:43.871396 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:43.872439 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:43.872599 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:43.873669 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:43.873678 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:43.873721 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:43.874405 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:43.874495 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:43.874519 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:43.874651 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:43.874832 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 02:12:43.875139 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":333,"threads_started":1}
I20260929 02:12:43.877735 30861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:43.878835 30861 log.cc:824] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:43.880141 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:12:43.881944 30861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9: No bootstrap required, opened a new log
I20260929 02:12:43.889212 30861 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc28eb514483451c81fc1f2106ef89a9" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42561 } }
I20260929 02:12:43.889503 30861 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:43.889573 30861 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc28eb514483451c81fc1f2106ef89a9, State: Initialized, Role: FOLLOWER
I20260929 02:12:43.890076 30861 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc28eb514483451c81fc1f2106ef89a9" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42561 } }
I20260929 02:12:43.890321 30861 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:43.890419 30861 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:43.890537 30861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:43.892409 30861 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc28eb514483451c81fc1f2106ef89a9" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42561 } }
I20260929 02:12:43.892814 30861 leader_election.cc:302] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc28eb514483451c81fc1f2106ef89a9; no voters:
I20260929 02:12:43.893306 30861 leader_election.cc:288] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:43.893479 30866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:43.894594 30866 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [term 1 LEADER]: Becoming Leader. State: Replica: bc28eb514483451c81fc1f2106ef89a9, State: Running, Role: LEADER
I20260929 02:12:43.895066 30866 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc28eb514483451c81fc1f2106ef89a9" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42561 } }
I20260929 02:12:43.895339 30861 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:43.897591 30868 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc28eb514483451c81fc1f2106ef89a9. Latest consensus state: current_term: 1 leader_uuid: "bc28eb514483451c81fc1f2106ef89a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc28eb514483451c81fc1f2106ef89a9" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42561 } } }
I20260929 02:12:43.897919 30868 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:43.897923 30867 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc28eb514483451c81fc1f2106ef89a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc28eb514483451c81fc1f2106ef89a9" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42561 } } }
I20260929 02:12:43.898696 30867 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:43.900182 30875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:43.905053 30875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:43.912920 30875 catalog_manager.cc:1347] Generated new cluster ID: 582aa04785e944c7a21115310653056b
I20260929 02:12:43.913018 30875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:43.923827 30875 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:43.925117 30875 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:43.932876 30875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc28eb514483451c81fc1f2106ef89a9: Generated new TSK 0
I20260929 02:12:43.933806 30875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:12:43.964804 30864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:43.956507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48832 (local address 127.24.216.190:42561)
0929 02:12:43.956710 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:43.956719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:43.957185 (+ 466us) server_negotiation.cc:407] Connection header received
0929 02:12:43.957419 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:43.957423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:43.957485 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:43.957599 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:12:43.958098 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:43.958864 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:43.960881 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:43.961366 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:43.961452 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:43.962014 (+ 562us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:12:43.964201 (+ 2187us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 02:12:43.964500 (+ 299us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48832: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 02:12:43.964923 30864 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48832: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 02:12:43.966997 30890 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:12:43.956488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:42561 (local address 127.0.0.1:48832)
0929 02:12:43.957081 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:43.957100 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:43.957238 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:43.957633 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:43.957638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:43.957650 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:12:43.957957 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:43.957968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:43.959006 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:43.959013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:43.960734 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:43.960749 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:43.961548 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:43.961554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:43.961739 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:43.961787 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:12:43.964416 (+ 2629us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 02:12:43.966549 (+ 2133us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.216.190:42561: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":1865,"thread_start_us":315,"threads_started":1}
I20260929 02:12:43.967916 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 30795
I20260929 02:12:43.968914 30856 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 02:12:43.980193 30891 webserver.cc:533] Webserver: sslize failed:
E20260929 02:12:43.980312 30891 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 02:12:44.059626 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 30795
2026-09-29T02:12:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (692 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T02:12:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:44Z Disabled control of system clock
I20260929 02:12:44.141455 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:35419
--webserver_interface=127.24.216.190
--webserver_port=0
--builtin_ntp_servers=127.24.216.148:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:35419
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/testchainca.cert with env {}
W20260929 02:12:44.452916 30896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:44.453222 30896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:44.453305 30896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:44.460502 30896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:44.460637 30896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:44.460682 30896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:44.460762 30896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:44.460826 30896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:12:44.460924 30896 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:12:44.460994 30896 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 02:12:44.470486 30896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:35419
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:35419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:44.471729 30896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:44.472970 30896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:44.484421 30901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:44.484391 30902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:44.484524 30904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:44.485155 30896 server_base.cc:1034] running on GCE node
I20260929 02:12:44.485873 30896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:44.487396 30896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:44.488623 30896 hybrid_clock.cc:648] HybridClock initialized: now 1790647964488585 us; error 27 us; skew 500 ppm
I20260929 02:12:44.488957 30896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:12:44.491315 30896 webserver.cc:460] Webserver started at http://127.24.216.190:43845/ using document root <none> and password file <none>
I20260929 02:12:44.491990 30896 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:44.492136 30896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:44.492443 30896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:44.495172 30896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "a16f533a588c452c83b51884eb8460df"
format_stamp: "Formatted at 2026-09-29 02:12:44 on dist-test-slave-9k0h"
server_key: "94cad124ac918db2d982bd68205751a3"
server_key_iv: "cb7f8fa3a6dbd1121f9497e88b20fba5"
server_key_version: "encryptionkey@0"
I20260929 02:12:44.495967 30896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "a16f533a588c452c83b51884eb8460df"
format_stamp: "Formatted at 2026-09-29 02:12:44 on dist-test-slave-9k0h"
server_key: "94cad124ac918db2d982bd68205751a3"
server_key_iv: "cb7f8fa3a6dbd1121f9497e88b20fba5"
server_key_version: "encryptionkey@0"
I20260929 02:12:44.500741 30896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:12:44.503942 30910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:44.505132 30896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:44.505321 30896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "a16f533a588c452c83b51884eb8460df"
format_stamp: "Formatted at 2026-09-29 02:12:44 on dist-test-slave-9k0h"
server_key: "94cad124ac918db2d982bd68205751a3"
server_key_iv: "cb7f8fa3a6dbd1121f9497e88b20fba5"
server_key_version: "encryptionkey@0"
I20260929 02:12:44.505543 30896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:44.532799 30896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:44.533689 30896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:44.533977 30896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:44.551724 30896 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:35419
I20260929 02:12:44.551744 30961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:35419 every 8 connection(s)
I20260929 02:12:44.553006 30896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:44.556595 30962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:44.558835 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 30896
I20260929 02:12:44.559017 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:44.559808 25442 external_mini_cluster.cc:1428] Setting key bee0fb0e86bba798f3a897420a7d7b89
I20260929 02:12:44.568552 30962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df: Bootstrap starting.
I20260929 02:12:44.572046 30962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:44.572904 30965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:44.561669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.24.216.190:35419)
0929 02:12:44.562679 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:44.562688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:44.562728 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:12:44.564196 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:44.564216 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:44.564650 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:44.564954 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:44.565508 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:44.566872 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:44.568694 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:44.569351 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:44.569477 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:44.570742 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:44.570749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:44.570762 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:12:44.571250 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:44.571342 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:12:44.571361 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:44.571444 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:44.571662 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 02:12:44.571980 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":363,"threads_started":1}
I20260929 02:12:44.573127 30962 log.cc:824] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:44.576192 30962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df: No bootstrap required, opened a new log
I20260929 02:12:44.576895 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:12:44.584566 30962 raft_consensus.cc:348] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16f533a588c452c83b51884eb8460df" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35419 } }
I20260929 02:12:44.584859 30962 raft_consensus.cc:374] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:44.584928 30962 raft_consensus.cc:729] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16f533a588c452c83b51884eb8460df, State: Initialized, Role: FOLLOWER
I20260929 02:12:44.585356 30962 consensus_queue.cc:260] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16f533a588c452c83b51884eb8460df" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35419 } }
I20260929 02:12:44.585530 30962 raft_consensus.cc:388] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:44.585618 30962 raft_consensus.cc:482] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:44.585731 30962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:44.587772 30962 raft_consensus.cc:504] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16f533a588c452c83b51884eb8460df" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35419 } }
I20260929 02:12:44.588219 30962 leader_election.cc:302] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a16f533a588c452c83b51884eb8460df; no voters:
I20260929 02:12:44.588778 30962 leader_election.cc:288] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:44.588897 30967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:44.589564 30967 raft_consensus.cc:686] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [term 1 LEADER]: Becoming Leader. State: Replica: a16f533a588c452c83b51884eb8460df, State: Running, Role: LEADER
I20260929 02:12:44.590209 30967 consensus_queue.cc:237] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16f533a588c452c83b51884eb8460df" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35419 } }
I20260929 02:12:44.590790 30962 sys_catalog.cc:564] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:44.593034 30968 sys_catalog.cc:455] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a16f533a588c452c83b51884eb8460df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16f533a588c452c83b51884eb8460df" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35419 } } }
I20260929 02:12:44.593004 30969 sys_catalog.cc:455] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [sys.catalog]: SysCatalogTable state changed. Reason: New leader a16f533a588c452c83b51884eb8460df. Latest consensus state: current_term: 1 leader_uuid: "a16f533a588c452c83b51884eb8460df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a16f533a588c452c83b51884eb8460df" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35419 } } }
I20260929 02:12:44.593438 30968 sys_catalog.cc:458] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:44.593448 30969 sys_catalog.cc:458] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:44.594425 30974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:44.599480 30974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:44.608428 30974 catalog_manager.cc:1347] Generated new cluster ID: 792b1f6a78744793a20ae7ece74fce84
I20260929 02:12:44.609632 30974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:44.618033 30974 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:44.619270 30974 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:44.627449 30974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a16f533a588c452c83b51884eb8460df: Generated new TSK 0
I20260929 02:12:44.628139 30974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:12:44.695345 30965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:44.649154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.24.216.190:35419)
0929 02:12:44.649402 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:44.649411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:44.649863 (+ 452us) server_negotiation.cc:407] Connection header received
0929 02:12:44.650058 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:44.650061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:44.650123 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:44.650293 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:12:44.650814 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:44.651482 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:44.653424 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:44.653858 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:44.653948 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:44.654481 (+ 533us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:12:44.694782 (+ 40301us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 02:12:44.695116 (+ 334us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58688: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 02:12:44.695497 30965 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58688: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 02:12:44.695815 30991 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:12:44.649126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:35419 (local address 127.0.0.1:58688)
0929 02:12:44.649740 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:44.649758 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:44.649918 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:44.650346 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:44.650350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:44.650363 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:12:44.650661 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:44.650674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:44.651617 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:44.651623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:44.653251 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:44.653267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:44.654016 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:44.654023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:44.654230 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:44.654274 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:12:44.695084 (+ 40810us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 02:12:44.695402 (+ 318us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.216.190:35419: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":325,"threads_started":1}
I20260929 02:12:44.697111 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 30896
I20260929 02:12:44.698341 30957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:44.790261 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 30896
2026-09-29T02:12:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (710 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (396 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (402 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:12:45 dist-test-slave-9k0h krb5kdc[31006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:12:45 dist-test-slave-9k0h krb5kdc[31006](info): set up 2 sockets
Sep 29 02:12:45 dist-test-slave-9k0h krb5kdc[31006](info): commencing operation
krb5kdc: starting...
W20260929 02:12:46.662966 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:12:46 dist-test-slave-9k0h krb5kdc[31006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:12:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:12:46.862073 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:43013
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:43013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:47.181121 31019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:47.181406 31019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:47.181491 31019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:47.188290 31019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:47.188385 31019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:47.188433 31019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:47.188508 31019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:47.188555 31019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:12:47.197993 31019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:43013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:43013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:47.199265 31019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:47.200459 31019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:47.212185 31025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:47.212252 31024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:47.212221 31027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:47.212803 31019 server_base.cc:1034] running on GCE node
I20260929 02:12:47.213636 31019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:47.215207 31019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:47.216415 31019 hybrid_clock.cc:648] HybridClock initialized: now 1790647967216374 us; error 38 us; skew 500 ppm
I20260929 02:12:47.216733 31019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:47 dist-test-slave-9k0h krb5kdc[31006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647967, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:47.220037 31019 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:12:47.222110 31019 webserver.cc:460] Webserver started at http://127.24.216.190:34037/ using document root <none> and password file <none>
I20260929 02:12:47.223061 31019 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:47.223230 31019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:47.223615 31019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:47.226673 31019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "7989c759db36496b8a868b68c235798a"
format_stamp: "Formatted at 2026-09-29 02:12:47 on dist-test-slave-9k0h"
server_key: "d9fae4c916dfcecaf3ed3f40d1fb1687"
server_key_iv: "d41fa6695e55e1c2ebad11222f8dbcc0"
server_key_version: "encryptionkey@0"
I20260929 02:12:47.227365 31019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "7989c759db36496b8a868b68c235798a"
format_stamp: "Formatted at 2026-09-29 02:12:47 on dist-test-slave-9k0h"
server_key: "d9fae4c916dfcecaf3ed3f40d1fb1687"
server_key_iv: "d41fa6695e55e1c2ebad11222f8dbcc0"
server_key_version: "encryptionkey@0"
I20260929 02:12:47.232125 31019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:12:47.235332 31034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:47.236567 31019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:47.236735 31019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "7989c759db36496b8a868b68c235798a"
format_stamp: "Formatted at 2026-09-29 02:12:47 on dist-test-slave-9k0h"
server_key: "d9fae4c916dfcecaf3ed3f40d1fb1687"
server_key_iv: "d41fa6695e55e1c2ebad11222f8dbcc0"
server_key_version: "encryptionkey@0"
I20260929 02:12:47.236901 31019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:47.272418 31019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:47.277077 31019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:47.277359 31019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:47.294328 31019 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:43013
I20260929 02:12:47.294363 31085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:43013 every 8 connection(s)
I20260929 02:12:47.295603 31019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:47.299224 31086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:47.302111 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 31019
I20260929 02:12:47.302372 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:47.303048 25442 external_mini_cluster.cc:1428] Setting key f3d0cee33cf5e4e0d9c7156afbd13cad
I20260929 02:12:47.311874 31086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a: Bootstrap starting.
I20260929 02:12:47.315218 31086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:47.316138 31086 log.cc:824] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:12:47 dist-test-slave-9k0h krb5kdc[31006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:47.318801 31086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a: No bootstrap required, opened a new log
I20260929 02:12:47.325266 31089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:47.304939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.24.216.190:43013)
0929 02:12:47.305733 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:47.305747 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:47.305795 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:12:47.307733 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:47.307765 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:47.308265 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:47.308691 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:47.309873 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:47.311806 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:47.313168 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:47.314245 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:47.314420 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:47.318255 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:47.318263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:47.318277 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:47.318319 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:47.321447 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:47.322102 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:47.322109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:47.322114 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:47.322335 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:47.322752 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:47.322762 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:47.322765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:47.323238 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:47.323562 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:47.323972 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 02:12:47.324354 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":329,"threads_started":1}
I20260929 02:12:47.326757 31086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7989c759db36496b8a868b68c235798a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43013 } }
I20260929 02:12:47.327090 31086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:47.327204 31086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7989c759db36496b8a868b68c235798a, State: Initialized, Role: FOLLOWER
I20260929 02:12:47.327771 31086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7989c759db36496b8a868b68c235798a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43013 } }
I20260929 02:12:47.328083 31086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:47.328219 31086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:47.328393 31086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:47.330454 31086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7989c759db36496b8a868b68c235798a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43013 } }
I20260929 02:12:47.330904 31086 leader_election.cc:302] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7989c759db36496b8a868b68c235798a; no voters:
I20260929 02:12:47.331427 31086 leader_election.cc:288] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:47.331566 31091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:47.332242 31091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [term 1 LEADER]: Becoming Leader. State: Replica: 7989c759db36496b8a868b68c235798a, State: Running, Role: LEADER
I20260929 02:12:47.332738 31091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7989c759db36496b8a868b68c235798a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43013 } }
I20260929 02:12:47.333333 31086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:47.336024 31093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7989c759db36496b8a868b68c235798a. Latest consensus state: current_term: 1 leader_uuid: "7989c759db36496b8a868b68c235798a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7989c759db36496b8a868b68c235798a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43013 } } }
I20260929 02:12:47.336478 31093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:47.336534 31092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7989c759db36496b8a868b68c235798a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7989c759db36496b8a868b68c235798a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43013 } } }
I20260929 02:12:47.340694 31092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:47.341427 31100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:47.345279 31100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:47.353534 31100 catalog_manager.cc:1347] Generated new cluster ID: 9708c6c417c54fc8a7411d16789419d3
I20260929 02:12:47.353639 31100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:47.379081 31100 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:47.380966 31100 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:47.395826 31100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7989c759db36496b8a868b68c235798a: Generated new TSK 0
I20260929 02:12:47.396857 31100 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:12:47.451923 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43013
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:47.765717 31112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:47.766023 31112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:47.766232 31112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:47.773161 31112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:47.773301 31112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:47.773473 31112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:12:47.782923 31112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:47.784152 31112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:47.785352 31112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:47.797526 31118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:47.797580 31117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:47.797614 31120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:47.798400 31112 server_base.cc:1034] running on GCE node
I20260929 02:12:47.798942 31112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:47.799839 31112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:47.801064 31112 hybrid_clock.cc:648] HybridClock initialized: now 1790647967800975 us; error 72 us; skew 500 ppm
I20260929 02:12:47.801363 31112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:47 dist-test-slave-9k0h krb5kdc[31006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647967, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:47.804440 31112 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:12:47.805763 31112 webserver.cc:460] Webserver started at http://127.24.216.129:42319/ using document root <none> and password file <none>
I20260929 02:12:47.806478 31112 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:47.806614 31112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:47.806910 31112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:47.809587 31112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "313d5eb217af428dbde6b6a995910165"
format_stamp: "Formatted at 2026-09-29 02:12:47 on dist-test-slave-9k0h"
server_key: "339c7b29440fe847713e0598d12bb48e"
server_key_iv: "319412a5fcaa05358cca4044abf787f7"
server_key_version: "encryptionkey@0"
I20260929 02:12:47.810403 31112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "313d5eb217af428dbde6b6a995910165"
format_stamp: "Formatted at 2026-09-29 02:12:47 on dist-test-slave-9k0h"
server_key: "339c7b29440fe847713e0598d12bb48e"
server_key_iv: "319412a5fcaa05358cca4044abf787f7"
server_key_version: "encryptionkey@0"
I20260929 02:12:47.815158 31112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:12:47.818358 31127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:47.819589 31112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:12:47.819758 31112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "313d5eb217af428dbde6b6a995910165"
format_stamp: "Formatted at 2026-09-29 02:12:47 on dist-test-slave-9k0h"
server_key: "339c7b29440fe847713e0598d12bb48e"
server_key_iv: "319412a5fcaa05358cca4044abf787f7"
server_key_version: "encryptionkey@0"
I20260929 02:12:47.819949 31112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:47.849166 31112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:47.853520 31112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:47.853793 31112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:47.854692 31112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:47.855865 31112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:47.855953 31112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:47.856083 31112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:47.856150 31112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:47.886677 31112 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:45385
I20260929 02:12:47.886734 31239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:45385 every 8 connection(s)
I20260929 02:12:47.887840 31112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:12:47.892668 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 31112
I20260929 02:12:47.892854 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:12:47.893420 25442 external_mini_cluster.cc:1428] Setting key 19b651036e25c26d5b142fb2fb019ea4
Sep 29 02:12:47 dist-test-slave-9k0h krb5kdc[31006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647967, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:47.909634 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:47.890326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:42339 (local address 127.24.216.190:43013)
0929 02:12:47.890797 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:47.890804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:47.892372 (+ 1568us) server_negotiation.cc:407] Connection header received
0929 02:12:47.894235 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:47.894240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:47.894327 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:47.894450 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:47.896280 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:47.896898 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:47.899379 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:47.899779 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:47.899875 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:47.904212 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:47.904217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:47.904221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:47.904247 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:47.906642 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:47.907501 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:47.907504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:47.907507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:47.907612 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:47.908068 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:47.908070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:47.908072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:47.908306 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:47.908405 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:47.909253 (+ 848us) server_negotiation.cc:299] Negotiation successful
0929 02:12:47.909399 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":214,"threads_started":1}
I20260929 02:12:47.911288 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:47.891040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43013 (local address 127.24.216.129:42339)
0929 02:12:47.892200 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:47.892247 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:47.893967 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:47.894659 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:47.894667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:47.895337 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:47.896122 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:47.896146 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:47.897050 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:47.897058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:47.899216 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:47.899233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:47.899999 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:47.900008 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:47.900385 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:47.901140 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:47.901173 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:47.904031 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:47.906777 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:47.906796 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:47.906814 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:47.907392 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:47.907708 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:47.907715 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:47.907721 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:47.907961 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:47.908426 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:47.908437 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:47.908888 (+ 451us) client_negotiation.cc:769] Sending connection context
0929 02:12:47.909168 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 02:12:47.909714 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":733,"threads_started":1}
I20260929 02:12:47.912845 31240 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43013
I20260929 02:12:47.913134 31240 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:47.913784 31240 heartbeater.cc:511] Master 127.24.216.190:43013 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
I20260929 02:12:47.915864 31051 ts_manager.cc:194] Registered new tserver with Master: 313d5eb217af428dbde6b6a995910165 (127.24.216.129:45385)
I20260929 02:12:47.917251 31051 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:42339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:12:47.941650 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43013
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:48.248713 31246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:48.248979 31246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:48.249145 31246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:48.255860 31246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:48.255950 31246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:48.256095 31246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:12:48.265202 31246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:48.266490 31246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:48.267814 31246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:48.280289 31251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:48.280376 31252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:48.280354 31254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:48.281210 31246 server_base.cc:1034] running on GCE node
I20260929 02:12:48.281759 31246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:48.282697 31246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:48.283917 31246 hybrid_clock.cc:648] HybridClock initialized: now 1790647968283885 us; error 51 us; skew 500 ppm
I20260929 02:12:48.284246 31246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:48 dist-test-slave-9k0h krb5kdc[31006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:48.287369 31246 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:12:48.288661 31246 webserver.cc:460] Webserver started at http://127.24.216.130:40999/ using document root <none> and password file <none>
I20260929 02:12:48.289301 31246 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:48.289441 31246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:48.289729 31246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:48.292555 31246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "7586c785eaa04934a32353ec8b09e5dc"
format_stamp: "Formatted at 2026-09-29 02:12:48 on dist-test-slave-9k0h"
server_key: "6c81c1c8db781e561d708f4d32114ce9"
server_key_iv: "3f4c19ab4d4074299d8a2f63bea1d86b"
server_key_version: "encryptionkey@0"
I20260929 02:12:48.293277 31246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "7586c785eaa04934a32353ec8b09e5dc"
format_stamp: "Formatted at 2026-09-29 02:12:48 on dist-test-slave-9k0h"
server_key: "6c81c1c8db781e561d708f4d32114ce9"
server_key_iv: "3f4c19ab4d4074299d8a2f63bea1d86b"
server_key_version: "encryptionkey@0"
I20260929 02:12:48.298183 31246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:12:48.301419 31261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:48.302744 31246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:48.302901 31246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "7586c785eaa04934a32353ec8b09e5dc"
format_stamp: "Formatted at 2026-09-29 02:12:48 on dist-test-slave-9k0h"
server_key: "6c81c1c8db781e561d708f4d32114ce9"
server_key_iv: "3f4c19ab4d4074299d8a2f63bea1d86b"
server_key_version: "encryptionkey@0"
I20260929 02:12:48.303089 31246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:48.336879 31246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:48.341197 31246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:48.341538 31246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:48.342475 31246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:48.343720 31246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:48.343823 31246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:48.343977 31246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:48.344048 31246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:48.375345 31246 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:37079
I20260929 02:12:48.375382 31373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:37079 every 8 connection(s)
I20260929 02:12:48.376613 31246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:12:48.379627 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 31246
I20260929 02:12:48.379782 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:12:48.380201 25442 external_mini_cluster.cc:1428] Setting key 46abebe2f152347c375aa567183b66c3
Sep 29 02:12:48 dist-test-slave-9k0h krb5kdc[31006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:48.397637 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:48.379034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:45035 (local address 127.24.216.190:43013)
0929 02:12:48.379208 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:48.379212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:48.380552 (+ 1340us) server_negotiation.cc:407] Connection header received
0929 02:12:48.382657 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:48.382664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:48.382748 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:48.382869 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:48.384229 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.385367 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.386982 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.387696 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.387847 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:48.391678 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:48.391682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:48.391685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:48.391714 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:48.394371 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:48.395238 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:48.395242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:48.395244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:48.395363 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:48.395859 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:48.395864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:48.395867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:48.396233 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:48.396353 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:48.397207 (+ 854us) server_negotiation.cc:299] Negotiation successful
0929 02:12:48.397370 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:12:48.399183 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:48.379549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43013 (local address 127.24.216.130:45035)
0929 02:12:48.380389 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:48.380434 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:48.382404 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:48.383053 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:48.383060 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:48.383539 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:48.384065 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:48.384081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.385510 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.385513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:48.386841 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:48.386853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.387896 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.387912 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:48.388094 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:48.388760 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:48.388786 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:48.391459 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:48.394496 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:48.394507 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:48.394526 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:48.395124 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:48.395471 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:48.395478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:48.395484 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:48.395723 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:48.396366 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:48.396376 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:48.396837 (+ 461us) client_negotiation.cc:769] Sending connection context
0929 02:12:48.397133 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 02:12:48.397684 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":412,"threads_started":1}
I20260929 02:12:48.400581 31374 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43013
I20260929 02:12:48.400888 31374 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:48.401562 31374 heartbeater.cc:511] Master 127.24.216.190:43013 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
I20260929 02:12:48.403007 31051 ts_manager.cc:194] Registered new tserver with Master: 7586c785eaa04934a32353ec8b09e5dc (127.24.216.130:37079)
I20260929 02:12:48.403843 31051 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:45035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:12:48.428627 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43013
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:48.744561 31379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:48.744818 31379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:48.744984 31379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:48.752135 31379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:48.752236 31379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:48.752409 31379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:12:48.761734 31379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:48.762965 31379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:48.764227 31379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:48.777005 31385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:48.777042 31384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:48.777009 31387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:48.777814 31379 server_base.cc:1034] running on GCE node
I20260929 02:12:48.778379 31379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:48.779196 31379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:48.780387 31379 hybrid_clock.cc:648] HybridClock initialized: now 1790647968780346 us; error 30 us; skew 500 ppm
I20260929 02:12:48.780683 31379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:48 dist-test-slave-9k0h krb5kdc[31006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:48.783779 31379 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:12:48.785148 31379 webserver.cc:460] Webserver started at http://127.24.216.131:38049/ using document root <none> and password file <none>
I20260929 02:12:48.785784 31379 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:48.785934 31379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:48.786285 31379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:48.789592 31379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "dc8c4206a558441c98085f845fb591ee"
format_stamp: "Formatted at 2026-09-29 02:12:48 on dist-test-slave-9k0h"
server_key: "4e34cd61d8776520933b5aeb3f6b14a3"
server_key_iv: "504b5ca059df10826776336ce06691ee"
server_key_version: "encryptionkey@0"
I20260929 02:12:48.790429 31379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "dc8c4206a558441c98085f845fb591ee"
format_stamp: "Formatted at 2026-09-29 02:12:48 on dist-test-slave-9k0h"
server_key: "4e34cd61d8776520933b5aeb3f6b14a3"
server_key_iv: "504b5ca059df10826776336ce06691ee"
server_key_version: "encryptionkey@0"
I20260929 02:12:48.795220 31379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:12:48.798563 31394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:48.799834 31379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:12:48.800010 31379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "dc8c4206a558441c98085f845fb591ee"
format_stamp: "Formatted at 2026-09-29 02:12:48 on dist-test-slave-9k0h"
server_key: "4e34cd61d8776520933b5aeb3f6b14a3"
server_key_iv: "504b5ca059df10826776336ce06691ee"
server_key_version: "encryptionkey@0"
I20260929 02:12:48.800204 31379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:48.816429 31379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:48.820649 31379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:48.820950 31379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:48.821841 31379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:48.823256 31379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:48.823374 31379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:48.823511 31379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:48.823583 31379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:48.855720 31379 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:45601
I20260929 02:12:48.855759 31506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:45601 every 8 connection(s)
I20260929 02:12:48.856992 31379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:12:48.866509 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 31379
I20260929 02:12:48.866681 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:12:48.867206 25442 external_mini_cluster.cc:1428] Setting key 641ee74bf25d4f0ab91170c115413e89
Sep 29 02:12:48 dist-test-slave-9k0h krb5kdc[31006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:12:48.875953 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:48.859483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:39991 (local address 127.24.216.190:43013)
0929 02:12:48.859657 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:48.859661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:48.860804 (+ 1143us) server_negotiation.cc:407] Connection header received
0929 02:12:48.862453 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:48.862457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:48.862535 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:48.862620 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:48.863947 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.864677 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.866049 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.866518 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.866609 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:48.870650 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:48.870655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:48.870658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:48.870686 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:48.873109 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:48.873759 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:48.873765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:48.873768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:48.873882 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:48.874341 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:48.874347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:48.874350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:48.874597 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:48.874753 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:48.875412 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 02:12:48.875624 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:12:48.876901 31509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:48.859976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43013 (local address 127.24.216.131:39991)
0929 02:12:48.860644 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:48.860686 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:48.862212 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:48.862800 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:48.862807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:48.863268 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:48.863788 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:48.863803 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.864800 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.864803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:48.865934 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:48.865948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.866781 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.866789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:48.867051 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:48.867743 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:48.867771 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:48.870464 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:48.873242 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:48.873250 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:48.873263 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:48.873643 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:48.873994 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:48.873999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:48.874003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:48.874223 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:48.874720 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:48.874728 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:48.875039 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 02:12:48.875321 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 02:12:48.875781 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":268,"threads_started":1}
I20260929 02:12:48.878238 31507 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43013
I20260929 02:12:48.878489 31507 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:48.879093 31507 heartbeater.cc:511] Master 127.24.216.190:43013 requested a full tablet report, sending...
I20260929 02:12:48.880394 31051 ts_manager.cc:194] Registered new tserver with Master: dc8c4206a558441c98085f845fb591ee (127.24.216.131:45601)
I20260929 02:12:48.881412 31051 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:39991
I20260929 02:12:48.882429 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/test-admin.keytab.
Sep 29 02:12:48 dist-test-slave-9k0h krb5kdc[31006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:48.909173 25442 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 02:12:48.913749 31514 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:12:48.911677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.24.216.129:45385)
0929 02:12:48.912218 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:48.912225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:48.912253 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:12:48.912320 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:48.912326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:48.912525 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:48.912646 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:48.913362 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53098: BlockingRecv error: recv got EOF from 127.0.0.1:53098 (error 108)
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":272,"threads_started":1}
W20260929 02:12:48.913938 25442 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/krb5cc' of type FILE
Sep 29 02:12:48 dist-test-slave-9k0h krb5kdc[31006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:48.915454 25442 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:12:48 dist-test-slave-9k0h krb5kdc[31006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:48.916728 25442 init.cc:295] Successfully reacquired a new kerberos TGT
I20260929 02:12:48.920854 31240 heartbeater.cc:503] Master 127.24.216.190:43013 was elected leader, sending a full tablet report...
Sep 29 02:12:48 dist-test-slave-9k0h krb5kdc[31006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:12:48.931908 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:48.918363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.24.216.129:45385)
0929 02:12:48.918527 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:48.918533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:48.918698 (+ 165us) server_negotiation.cc:407] Connection header received
0929 02:12:48.918917 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:48.918921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:48.918972 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:48.919069 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:48.920049 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.921232 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.922827 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.923538 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.923633 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:48.927035 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:48.927044 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:48.927056 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:48.927082 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:48.929510 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:48.930080 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:48.930086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:48.930088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:48.930259 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:48.930563 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:48.930566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:48.930568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:48.930893 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:48.931117 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:48.931398 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 02:12:48.931659 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 02:12:48.937157 31514 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:12:48.935496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.24.216.129:45385)
0929 02:12:48.935687 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:48.935694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:48.935945 (+ 251us) server_negotiation.cc:407] Connection header received
0929 02:12:48.936101 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:48.936106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:48.936164 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:48.936256 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:48.936910 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53118: BlockingRecv error: recv got EOF from 127.0.0.1:53118 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 02:12:48.937536 25442 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/krb5cc)
Sep 29 02:12:48 dist-test-slave-9k0h krb5kdc[31006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:48.939006 25442 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:12:48 dist-test-slave-9k0h krb5kdc[31006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:48.940322 25442 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 02:12:48 dist-test-slave-9k0h krb5kdc[31006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:12:48.953331 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:48.941557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.24.216.129:45385)
0929 02:12:48.941721 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:48.941729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:48.941999 (+ 270us) server_negotiation.cc:407] Connection header received
0929 02:12:48.942175 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:48.942180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:48.942240 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:48.942355 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:48.943220 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.943861 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.945001 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.945400 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.945549 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:48.948701 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:48.948707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:48.948711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:48.948748 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:48.951117 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:48.951679 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:48.951686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:48.951689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:48.951799 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:48.952144 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:48.952150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:48.952153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:48.952388 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:48.952618 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:48.952922 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 02:12:48.953097 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 02:12:48.957043 31514 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:12:48.955492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.24.216.129:45385)
0929 02:12:48.955664 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:48.955670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:48.955928 (+ 258us) server_negotiation.cc:407] Connection header received
0929 02:12:48.956081 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:48.956085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:48.956158 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:48.956246 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:48.956838 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53132: BlockingRecv error: recv got EOF from 127.0.0.1:53132 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 02:12:48.957237 25442 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790647920678408-25442-0/krb5kdc/krb5cc' of type FILE
Sep 29 02:12:48 dist-test-slave-9k0h krb5kdc[31006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:48.958581 25442 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:12:48 dist-test-slave-9k0h krb5kdc[31006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:48.959771 25442 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 02:12:48 dist-test-slave-9k0h krb5kdc[31006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:12:48.972406 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:48.960867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53146 (local address 127.24.216.129:45385)
0929 02:12:48.961057 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:48.961062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:48.961298 (+ 236us) server_negotiation.cc:407] Connection header received
0929 02:12:48.961390 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:48.961393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:48.961436 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:48.961503 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:48.962426 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.963100 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.964135 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.964496 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.964577 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:48.967617 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:48.967621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:48.967624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:48.967649 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:48.970025 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:48.970638 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:48.970642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:48.970644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:48.970756 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:48.971138 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:48.971142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:48.971144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:48.971370 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:48.971680 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:48.971954 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 02:12:48.972142 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:12:48.973788 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 31112
I20260929 02:12:48.984081 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:48.974447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.24.216.129:45385)
0929 02:12:48.974571 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:48.974574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:48.974940 (+ 366us) server_negotiation.cc:407] Connection header received
0929 02:12:48.975039 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:48.975042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:48.975087 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:48.975157 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:48.975945 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.976564 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.977693 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:48.978058 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:48.978182 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:48.979895 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:48.979899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:48.979902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:48.979927 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:48.982094 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:48.982661 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:48.982665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:48.982667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:48.982783 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:48.983083 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:48.983086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:48.983088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:48.983284 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:48.983453 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:48.983696 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 02:12:48.983839 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:12:48.984566 31235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:49.065255 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 31112
I20260929 02:12:49.081779 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 31246
Sep 29 02:12:49 dist-test-slave-9k0h krb5kdc[31006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:12:49.095831 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:49.082639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.24.216.130:37079)
0929 02:12:49.083150 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:49.083158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:49.083183 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:12:49.083243 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:49.083249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:49.083458 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:49.083590 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:49.084457 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:49.085438 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:49.086708 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:49.087219 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:49.087314 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:49.090887 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:49.090897 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:49.090910 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:49.090938 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:49.093427 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:49.093983 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:49.093990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:49.093992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:49.094098 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:49.094604 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:49.094607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:49.094609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:49.094913 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:49.095125 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:49.095354 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 02:12:49.095526 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":273,"threads_started":1}
I20260929 02:12:49.097023 31369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:49.177057 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 31246
I20260929 02:12:49.192713 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 31379
Sep 29 02:12:49 dist-test-slave-9k0h krb5kdc[31006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:12:49.208009 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:49.193576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35870 (local address 127.24.216.131:45601)
0929 02:12:49.194144 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:49.194203 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:49.194244 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:12:49.194309 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:49.194315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:49.194556 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:49.194716 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:49.195747 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:49.196753 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:49.197970 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:49.198672 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:49.198810 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:49.202648 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:49.202658 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:49.202671 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:49.202706 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:49.205369 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:49.206029 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:49.206036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:49.206038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:49.206202 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:49.206564 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:49.206572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:49.206575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:49.206894 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:49.207207 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:49.207467 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 02:12:49.207700 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":274,"threads_started":1}
I20260929 02:12:49.209219 31502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:49.292234 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 31379
I20260929 02:12:49.307607 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 31019
I20260929 02:12:49.308655 31081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:49.408258 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 31019
2026-09-29T02:12:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3825 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:12:49 dist-test-slave-9k0h krb5kdc[31534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:12:49 dist-test-slave-9k0h krb5kdc[31534](info): set up 2 sockets
Sep 29 02:12:49 dist-test-slave-9k0h krb5kdc[31534](info): commencing operation
krb5kdc: starting...
W20260929 02:12:50.478866 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.005s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:12:50 dist-test-slave-9k0h krb5kdc[31534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:12:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:12:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.190.keytab.
Entry for principal oryx/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.190.keytab.
I20260929 02:12:50.675731 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:39637
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.190.keytab
--principal=oryx/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:39637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:50.987627 31547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:50.987967 31547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:50.988155 31547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:50.995229 31547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:50.995373 31547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:50.995512 31547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:50.995637 31547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:50.995715 31547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:12:51.005144 31547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:39637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.190.keytab
--principal=oryx/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:39637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:51.006541 31547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:51.007819 31547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:51.019261 31552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:51.019333 31555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:51.019353 31553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:51.020155 31547 server_base.cc:1034] running on GCE node
I20260929 02:12:51.020829 31547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:51.022035 31547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:51.023227 31547 hybrid_clock.cc:648] HybridClock initialized: now 1790647971023188 us; error 39 us; skew 500 ppm
I20260929 02:12:51.023517 31547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:51 dist-test-slave-9k0h krb5kdc[31534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647971, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:51.026454 31547 init.cc:377] Logged in from keytab as oryx/127.24.216.190@KRBTEST.COM (short username oryx)
I20260929 02:12:51.027809 31547 webserver.cc:460] Webserver started at http://127.24.216.190:33415/ using document root <none> and password file <none>
I20260929 02:12:51.028429 31547 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:51.028522 31547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:51.028781 31547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:51.031703 31547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "75301924e06d4b8682d589278c402ce8"
format_stamp: "Formatted at 2026-09-29 02:12:51 on dist-test-slave-9k0h"
server_key: "86338b8501fc6f18eb4fd8aedd6ec07e"
server_key_iv: "5371f5747d854562a64b9b424ef52fdc"
server_key_version: "encryptionkey@0"
I20260929 02:12:51.032411 31547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "75301924e06d4b8682d589278c402ce8"
format_stamp: "Formatted at 2026-09-29 02:12:51 on dist-test-slave-9k0h"
server_key: "86338b8501fc6f18eb4fd8aedd6ec07e"
server_key_iv: "5371f5747d854562a64b9b424ef52fdc"
server_key_version: "encryptionkey@0"
I20260929 02:12:51.037153 31547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 02:12:51.040367 31562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:51.041612 31547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:12:51.041747 31547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "75301924e06d4b8682d589278c402ce8"
format_stamp: "Formatted at 2026-09-29 02:12:51 on dist-test-slave-9k0h"
server_key: "86338b8501fc6f18eb4fd8aedd6ec07e"
server_key_iv: "5371f5747d854562a64b9b424ef52fdc"
server_key_version: "encryptionkey@0"
I20260929 02:12:51.041895 31547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:51.057142 31547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:51.060774 31547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:51.061010 31547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:51.078665 31547 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:39637
I20260929 02:12:51.078689 31613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:39637 every 8 connection(s)
I20260929 02:12:51.080009 31547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:12:51.083551 31614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:51.084941 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 31547
I20260929 02:12:51.085146 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:12:51.086000 25442 external_mini_cluster.cc:1428] Setting key ac19a1af2bd64532c165f284f744ea54
I20260929 02:12:51.096125 31614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8: Bootstrap starting.
Sep 29 02:12:51 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
I20260929 02:12:51.099210 31614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:51.100219 31614 log.cc:824] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:51.103060 31614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8: No bootstrap required, opened a new log
I20260929 02:12:51.106725 31617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:51.088055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44426 (local address 127.24.216.190:39637)
0929 02:12:51.088823 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:51.088838 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:51.088881 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:12:51.090455 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:51.090482 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:51.090862 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:51.091258 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:51.092622 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:51.093900 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:51.095179 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:51.095813 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:51.096002 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:51.099872 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:51.099879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:51.099892 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:51.099924 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:51.102933 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:51.103634 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:51.103649 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:51.103655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:51.103816 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:51.104177 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:51.104184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:51.104187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:51.104683 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:51.105045 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:51.105375 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 02:12:51.105759 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":323,"threads_started":1}
I20260929 02:12:51.110718 31614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } }
I20260929 02:12:51.111070 31614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:51.111183 31614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75301924e06d4b8682d589278c402ce8, State: Initialized, Role: FOLLOWER
I20260929 02:12:51.111754 31614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } }
I20260929 02:12:51.111982 31614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:12:51.112121 31614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:12:51.112283 31614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:51.114507 31614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } }
I20260929 02:12:51.115052 31614 leader_election.cc:302] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75301924e06d4b8682d589278c402ce8; no voters:
I20260929 02:12:51.115757 31614 leader_election.cc:288] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:12:51.115881 31619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:51.116622 31619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [term 1 LEADER]: Becoming Leader. State: Replica: 75301924e06d4b8682d589278c402ce8, State: Running, Role: LEADER
I20260929 02:12:51.117146 31619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } }
I20260929 02:12:51.117827 31614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:51.120121 31620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } }
I20260929 02:12:51.120502 31620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:51.120421 31621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75301924e06d4b8682d589278c402ce8. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } }
I20260929 02:12:51.120725 31621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:51.122596 31626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:51.127084 31626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:51.135100 31626 catalog_manager.cc:1347] Generated new cluster ID: c451746408114b63bc62a23d8e73de81
I20260929 02:12:51.135205 31626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:51.157917 31626 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:12:51.159874 31626 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:51.171123 31626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8: Generated new TSK 0
I20260929 02:12:51.172222 31626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.129.keytab.
Entry for principal oryx/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.129.keytab.
I20260929 02:12:51.233536 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39637
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.129.keytab
--principal=oryx/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:51.572024 31640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:51.572270 31640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:51.572359 31640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:51.579645 31640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:51.579741 31640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:51.579850 31640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:12:51.588699 31640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.129.keytab
--principal=oryx/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:51.589787 31640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:51.590996 31640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:51.603458 31645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:51.603544 31648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:51.603581 31646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:51.604436 31640 server_base.cc:1034] running on GCE node
I20260929 02:12:51.605015 31640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:51.605904 31640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:51.607123 31640 hybrid_clock.cc:648] HybridClock initialized: now 1790647971607036 us; error 101 us; skew 500 ppm
I20260929 02:12:51.607374 31640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:51 dist-test-slave-9k0h krb5kdc[31534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647971, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:51.610368 31640 init.cc:377] Logged in from keytab as oryx/127.24.216.129@KRBTEST.COM (short username oryx)
I20260929 02:12:51.611681 31640 webserver.cc:460] Webserver started at http://127.24.216.129:36301/ using document root <none> and password file <none>
I20260929 02:12:51.612310 31640 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:51.612447 31640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:51.612727 31640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:51.615584 31640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "1762517f34e643f5a4d2ad8adf685336"
format_stamp: "Formatted at 2026-09-29 02:12:51 on dist-test-slave-9k0h"
server_key: "464dd62bdc29cf491642e1e9defff8cf"
server_key_iv: "433237ab8629721348363cd8a2536abf"
server_key_version: "encryptionkey@0"
I20260929 02:12:51.616317 31640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "1762517f34e643f5a4d2ad8adf685336"
format_stamp: "Formatted at 2026-09-29 02:12:51 on dist-test-slave-9k0h"
server_key: "464dd62bdc29cf491642e1e9defff8cf"
server_key_iv: "433237ab8629721348363cd8a2536abf"
server_key_version: "encryptionkey@0"
I20260929 02:12:51.621161 31640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:12:51.624413 31655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:51.625545 31640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:51.625734 31640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "1762517f34e643f5a4d2ad8adf685336"
format_stamp: "Formatted at 2026-09-29 02:12:51 on dist-test-slave-9k0h"
server_key: "464dd62bdc29cf491642e1e9defff8cf"
server_key_iv: "433237ab8629721348363cd8a2536abf"
server_key_version: "encryptionkey@0"
I20260929 02:12:51.625933 31640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:51.639719 31640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:51.643744 31640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:51.644024 31640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:51.644867 31640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:51.646111 31640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:51.646252 31640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:51.646394 31640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:51.646504 31640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:51.678131 31640 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:38757
I20260929 02:12:51.678226 31767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:38757 every 8 connection(s)
I20260929 02:12:51.679522 31640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:12:51.683142 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 31640
I20260929 02:12:51.683377 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:12:51.684290 25442 external_mini_cluster.cc:1428] Setting key 6c67fc01f603e5633c68cbc3f4d5d2e5
Sep 29 02:12:51 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647971, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
I20260929 02:12:51.702811 31770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:51.682026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:59431 (local address 127.24.216.190:39637)
0929 02:12:51.682626 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:51.682634 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:51.683616 (+ 982us) server_negotiation.cc:407] Connection header received
0929 02:12:51.686096 (+ 2480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:51.686101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:51.686203 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:51.686313 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:51.688361 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:51.689077 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:51.691630 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:51.692040 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:51.692133 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:51.696213 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:51.696222 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:51.696229 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:51.696269 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:51.699698 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:51.700395 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:51.700404 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:51.700410 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:51.700548 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:51.700941 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:51.700948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:51.700953 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:51.701303 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:51.701454 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:51.702066 (+ 612us) server_negotiation.cc:299] Negotiation successful
0929 02:12:51.702335 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":333,"threads_started":1}
I20260929 02:12:51.703392 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:51.682564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.129:59431)
0929 02:12:51.683419 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:51.683489 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:51.685810 (+ 2321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:51.686586 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:51.686598 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:51.687330 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:51.688170 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:51.688198 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:51.689249 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:51.689257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:51.691482 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:51.691500 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:51.692267 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:51.692276 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:51.692628 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:51.693360 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:51.693395 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:51.696005 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:51.699856 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:51.699862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:51.699873 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:51.700260 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:51.700651 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:51.700654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:51.700656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:51.700828 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:51.701417 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:51.701422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:51.701704 (+ 282us) client_negotiation.cc:769] Sending connection context
0929 02:12:51.701918 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 02:12:51.702336 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":306,"threads_started":1}
I20260929 02:12:51.704845 31768 heartbeater.cc:347] Connected to a master server at 127.24.216.190:39637
I20260929 02:12:51.705128 31768 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:51.705799 31768 heartbeater.cc:511] Master 127.24.216.190:39637 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.24.216.130@KRBTEST.COM; defaulting to no policy
I20260929 02:12:51.707908 31579 ts_manager.cc:194] Registered new tserver with Master: 1762517f34e643f5a4d2ad8adf685336 (127.24.216.129:38757)
I20260929 02:12:51.709508 31579 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:59431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.130.keytab.
Entry for principal oryx/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.130.keytab.
I20260929 02:12:51.733490 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39637
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.130.keytab
--principal=oryx/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:52.041039 31774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:52.041292 31774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:52.041381 31774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:52.048326 31774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:52.048417 31774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:52.048520 31774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:12:52.057631 31774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.130.keytab
--principal=oryx/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:52.058753 31774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:52.060024 31774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:52.072465 31779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:52.072490 31782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:52.072520 31780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:52.073513 31774 server_base.cc:1034] running on GCE node
I20260929 02:12:52.074122 31774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:52.075021 31774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:52.076241 31774 hybrid_clock.cc:648] HybridClock initialized: now 1790647972076184 us; error 48 us; skew 500 ppm
I20260929 02:12:52.076555 31774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:52 dist-test-slave-9k0h krb5kdc[31534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647972, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:52.079738 31774 init.cc:377] Logged in from keytab as oryx/127.24.216.130@KRBTEST.COM (short username oryx)
I20260929 02:12:52.081195 31774 webserver.cc:460] Webserver started at http://127.24.216.130:36047/ using document root <none> and password file <none>
I20260929 02:12:52.081867 31774 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:52.082008 31774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:52.082358 31774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:52.085037 31774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "45cffc9111ec424881246da5734a8573"
format_stamp: "Formatted at 2026-09-29 02:12:52 on dist-test-slave-9k0h"
server_key: "bf13744c295af70bf9390c572d10ea74"
server_key_iv: "9c2686c32b71e6e6ae2373daadc581cf"
server_key_version: "encryptionkey@0"
I20260929 02:12:52.085774 31774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "45cffc9111ec424881246da5734a8573"
format_stamp: "Formatted at 2026-09-29 02:12:52 on dist-test-slave-9k0h"
server_key: "bf13744c295af70bf9390c572d10ea74"
server_key_iv: "9c2686c32b71e6e6ae2373daadc581cf"
server_key_version: "encryptionkey@0"
I20260929 02:12:52.090648 31774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260929 02:12:52.093863 31789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:52.095062 31774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:52.095217 31774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "45cffc9111ec424881246da5734a8573"
format_stamp: "Formatted at 2026-09-29 02:12:52 on dist-test-slave-9k0h"
server_key: "bf13744c295af70bf9390c572d10ea74"
server_key_iv: "9c2686c32b71e6e6ae2373daadc581cf"
server_key_version: "encryptionkey@0"
I20260929 02:12:52.095402 31774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:52.114537 31774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:52.118191 31774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:52.118462 31774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:52.119419 31774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:52.120678 31774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:52.120776 31774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:52.120926 31774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:52.120995 31774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:52.153415 31774 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:36219
I20260929 02:12:52.153455 31901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:36219 every 8 connection(s)
I20260929 02:12:52.154937 31774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:12:52.162073 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 31774
I20260929 02:12:52.162281 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:12:52.162783 25442 external_mini_cluster.cc:1428] Setting key 95395e660370dd21d313267d073ac05e
Sep 29 02:12:52 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647972, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
I20260929 02:12:52.175678 31770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:52.157578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:41463 (local address 127.24.216.190:39637)
0929 02:12:52.157752 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:52.157756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:52.158867 (+ 1111us) server_negotiation.cc:407] Connection header received
0929 02:12:52.160629 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:52.160635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:52.160726 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:52.160843 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:52.162524 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:52.163307 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:52.165256 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:52.165636 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:52.165730 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:52.170143 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:52.170187 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:52.170195 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:52.170238 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:52.172868 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:52.173527 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:52.173538 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:52.173542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:52.173654 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:52.174043 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:52.174048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:52.174051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:52.174347 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:52.174491 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:52.175128 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 02:12:52.175337 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:12:52.176651 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:52.158022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.130:41463)
0929 02:12:52.158710 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:52.158759 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:52.160360 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:52.161061 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:52.161070 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:52.161548 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:52.162360 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:52.162385 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:52.163475 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:52.163479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:52.165103 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:52.165117 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:52.165865 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:52.165874 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:52.166242 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:52.166982 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:52.167019 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:52.169922 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:52.173000 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:52.173010 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:52.173028 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:52.173406 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:52.173761 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:52.173766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:52.173770 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:52.173931 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:52.174466 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:52.174474 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:52.174794 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 02:12:52.175039 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 02:12:52.175533 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":285,"threads_started":1}
I20260929 02:12:52.178181 31902 heartbeater.cc:347] Connected to a master server at 127.24.216.190:39637
I20260929 02:12:52.178478 31902 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:52.179178 31902 heartbeater.cc:511] Master 127.24.216.190:39637 requested a full tablet report, sending...
I20260929 02:12:52.180781 31579 ts_manager.cc:194] Registered new tserver with Master: 45cffc9111ec424881246da5734a8573 (127.24.216.130:36219)
I20260929 02:12:52.182120 31579 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:41463
WARNING: no policy specified for oryx/127.24.216.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.131.keytab.
Entry for principal oryx/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.131.keytab.
I20260929 02:12:52.209859 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39637
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.131.keytab
--principal=oryx/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:52.525514 31907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:52.525759 31907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:52.525939 31907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:52.533105 31907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:52.533236 31907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:52.533403 31907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:12:52.543370 31907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.131.keytab
--principal=oryx/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:39637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:52.544632 31907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:52.546010 31907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:52.559347 31912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:52.559302 31913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:52.559335 31915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:52.560070 31907 server_base.cc:1034] running on GCE node
I20260929 02:12:52.560683 31907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:52.561585 31907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:52.562808 31907 hybrid_clock.cc:648] HybridClock initialized: now 1790647972562745 us; error 55 us; skew 500 ppm
I20260929 02:12:52.563107 31907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:52 dist-test-slave-9k0h krb5kdc[31534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647972, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:52.566237 31907 init.cc:377] Logged in from keytab as oryx/127.24.216.131@KRBTEST.COM (short username oryx)
I20260929 02:12:52.567677 31907 webserver.cc:460] Webserver started at http://127.24.216.131:45517/ using document root <none> and password file <none>
I20260929 02:12:52.568364 31907 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:52.568506 31907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:52.568840 31907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:52.571852 31907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "979224aefeeb448287acefed22a220c9"
format_stamp: "Formatted at 2026-09-29 02:12:52 on dist-test-slave-9k0h"
server_key: "c483cd419fb98047fb98982756f358b5"
server_key_iv: "e699d5685699a70dd213fab4b1349986"
server_key_version: "encryptionkey@0"
I20260929 02:12:52.572636 31907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "979224aefeeb448287acefed22a220c9"
format_stamp: "Formatted at 2026-09-29 02:12:52 on dist-test-slave-9k0h"
server_key: "c483cd419fb98047fb98982756f358b5"
server_key_iv: "e699d5685699a70dd213fab4b1349986"
server_key_version: "encryptionkey@0"
I20260929 02:12:52.577834 31907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 02:12:52.581275 31922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:52.582664 31907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:52.582828 31907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "979224aefeeb448287acefed22a220c9"
format_stamp: "Formatted at 2026-09-29 02:12:52 on dist-test-slave-9k0h"
server_key: "c483cd419fb98047fb98982756f358b5"
server_key_iv: "e699d5685699a70dd213fab4b1349986"
server_key_version: "encryptionkey@0"
I20260929 02:12:52.583029 31907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:52.633666 31907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:52.638273 31907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:52.638593 31907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:52.639616 31907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:52.640993 31907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:52.641077 31907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:52.641264 31907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:52.641321 31907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:52.675033 31907 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:39063
I20260929 02:12:52.675069 32034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:39063 every 8 connection(s)
I20260929 02:12:52.676412 31907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:12:52.679677 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 31907
I20260929 02:12:52.679854 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:12:52.680503 25442 external_mini_cluster.cc:1428] Setting key eea9e76bb593aa6dd1b2b20d7cd9729f
Sep 29 02:12:52 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647972, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
I20260929 02:12:52.698859 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:52.679216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:42691 (local address 127.24.216.190:39637)
0929 02:12:52.685258 (+ 6042us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:52.685262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:52.685275 (+ 13us) server_negotiation.cc:407] Connection header received
0929 02:12:52.685322 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:52.685325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:52.685380 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:52.685499 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:52.687049 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:52.687942 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:52.689455 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:52.689871 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:52.690003 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:52.693185 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:52.693191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:52.693196 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:52.693234 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:52.696079 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:52.696913 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:52.696917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:52.696919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:52.697030 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:52.697371 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:52.697374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:52.697375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:52.697603 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:52.697709 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:52.698334 (+ 625us) server_negotiation.cc:299] Negotiation successful
0929 02:12:52.698593 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5956,"thread_start_us":5859,"threads_started":1}
I20260929 02:12:52.699779 32038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:52.679847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.131:42691)
0929 02:12:52.682753 (+ 2906us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:52.682811 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:52.684777 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:52.685723 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:52.685730 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:52.686248 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:52.686887 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:52.686904 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:52.688082 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:52.688092 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:52.689310 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:52.689321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:52.690092 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:52.690099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:52.690333 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:52.690953 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:52.690973 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:52.692985 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:52.696218 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:52.696224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:52.696234 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:52.696803 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:52.697116 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:52.697119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:52.697121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:52.697280 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:52.697710 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:52.697715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:52.697979 (+ 264us) client_negotiation.cc:769] Sending connection context
0929 02:12:52.698248 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 02:12:52.698653 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2630,"thread_start_us":2419,"threads_started":1}
I20260929 02:12:52.701030 32035 heartbeater.cc:347] Connected to a master server at 127.24.216.190:39637
I20260929 02:12:52.701330 32035 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:52.701994 32035 heartbeater.cc:511] Master 127.24.216.190:39637 requested a full tablet report, sending...
I20260929 02:12:52.703336 31579 ts_manager.cc:194] Registered new tserver with Master: 979224aefeeb448287acefed22a220c9 (127.24.216.131:39063)
I20260929 02:12:52.704102 31579 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:42691
I20260929 02:12:52.708482 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:12:52.713102 31768 heartbeater.cc:503] Master 127.24.216.190:39637 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.24.216.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.216.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.216.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.189.keytab.
Entry for principal oryx/127.24.216.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.189.keytab.
I20260929 02:12:52.753377 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.189:34095
--webserver_interface=127.24.216.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.189.keytab
--principal=oryx/127.24.216.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:39637,127.24.216.189:34095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:53.061298 32041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:53.061573 32041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:53.061666 32041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:53.068602 32041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:53.068724 32041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:53.068769 32041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:53.068840 32041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:53.068902 32041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:12:53.077975 32041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:39637,127.24.216.189:34095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.189.keytab
--principal=oryx/127.24.216.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.189:34095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.216.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
W20260929 02:12:53.078419 32041 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.24.216.190:39637,127.24.216.189:34095'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 02:12:53.079459 32041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:53.080796 32041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:53.092216 32049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:53.092299 32047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:53.092319 32046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:53.092958 32041 server_base.cc:1034] running on GCE node
I20260929 02:12:53.093597 32041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:53.094961 32041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:53.096163 32041 hybrid_clock.cc:648] HybridClock initialized: now 1790647973096108 us; error 28 us; skew 500 ppm
I20260929 02:12:53.096415 32041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:53 dist-test-slave-9k0h krb5kdc[31534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647973, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:53.099325 32041 init.cc:377] Logged in from keytab as oryx/127.24.216.189@KRBTEST.COM (short username oryx)
I20260929 02:12:53.100747 32041 webserver.cc:460] Webserver started at http://127.24.216.189:46759/ using document root <none> and password file <none>
I20260929 02:12:53.101425 32041 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:53.101555 32041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:53.101861 32041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:53.104744 32041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/data/instance:
uuid: "c752a8f94421423dbdd6388c0a1d8845"
format_stamp: "Formatted at 2026-09-29 02:12:53 on dist-test-slave-9k0h"
server_key: "3b577079c651d93c4595d912ae84082f"
server_key_iv: "4a43fae91609bac8f833b2c2c35d91ea"
server_key_version: "encryptionkey@0"
I20260929 02:12:53.105504 32041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/wal/instance:
uuid: "c752a8f94421423dbdd6388c0a1d8845"
format_stamp: "Formatted at 2026-09-29 02:12:53 on dist-test-slave-9k0h"
server_key: "3b577079c651d93c4595d912ae84082f"
server_key_iv: "4a43fae91609bac8f833b2c2c35d91ea"
server_key_version: "encryptionkey@0"
I20260929 02:12:53.110472 32041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 02:12:53.113618 32056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:53.114894 32041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:53.115070 32041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/wal
uuid: "c752a8f94421423dbdd6388c0a1d8845"
format_stamp: "Formatted at 2026-09-29 02:12:53 on dist-test-slave-9k0h"
server_key: "3b577079c651d93c4595d912ae84082f"
server_key_iv: "4a43fae91609bac8f833b2c2c35d91ea"
server_key_version: "encryptionkey@0"
I20260929 02:12:53.115280 32041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:53.131201 32041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:53.134789 32041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:53.135036 32041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:53.153043 32041 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.189:34095
I20260929 02:12:53.153087 32107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.189:34095 every 8 connection(s)
I20260929 02:12:53.154598 32041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/data/info.pb
I20260929 02:12:53.158916 32108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:53.161338 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 32041
I20260929 02:12:53.165843 32108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:12:53 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.216.189@KRBTEST.COM
Sep 29 02:12:53 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647973, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.189@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
I20260929 02:12:53.184620 31902 heartbeater.cc:503] Master 127.24.216.190:39637 was elected leader, sending a full tablet report...
I20260929 02:12:53.184811 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.168661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44428 (local address 127.24.216.190:39637)
0929 02:12:53.168824 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.168828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.169599 (+ 771us) server_negotiation.cc:407] Connection header received
0929 02:12:53.169800 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.169808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.169888 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.169996 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.171396 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.172364 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.174130 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.174623 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.174734 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.178299 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.178308 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.178314 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.178358 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.181593 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.182384 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.182391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.182396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.182511 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.182956 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.182960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.182963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.183337 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.183467 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.184074 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.184506 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:12:53.185648 32111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.163355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37432 (local address 127.24.216.189:34095)
0929 02:12:53.164189 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.164200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.164239 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:12:53.165792 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.165814 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.166310 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.166633 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.168441 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.170289 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.171908 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.172858 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.172993 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.177831 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.177839 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.177850 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.177878 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.181333 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.182043 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.182055 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.182058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.182285 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.182691 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.182694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.182696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.183312 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.183597 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.183951 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.184412 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":643,"thread_start_us":414,"threads_started":1}
I20260929 02:12:53.185686 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.168768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:44428)
0929 02:12:53.169444 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:53.169501 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:53.169666 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:53.170045 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:53.170051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:53.170908 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:53.171208 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.171220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.172522 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.172525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.173971 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.173986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.174858 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.174866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.175071 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.175718 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:53.175747 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:53.178071 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:53.181707 (+ 3636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.181717 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.181728 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.182278 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.182615 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.182623 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.182628 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.182833 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.183456 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:53.183463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:53.183670 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 02:12:53.183955 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 02:12:53.184543 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":153,"thread_start_us":309,"threads_started":1}
I20260929 02:12:53.187738 32108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:12:53 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647973, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.189@KRBTEST.COM for oryx/127.24.216.189@KRBTEST.COM
I20260929 02:12:53.198608 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.188373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:34095 (local address 127.0.0.1:37448)
0929 02:12:53.188537 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:53.188550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:53.188619 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:53.188968 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:53.188972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:53.189301 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:53.189416 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.189423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.190402 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.190407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.191257 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.191267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.191896 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.191913 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.192101 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.192517 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:53.192531 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:53.194325 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:53.196651 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.196655 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.196657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.196989 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.197294 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.197297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.197299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.197412 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.197827 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:53.197830 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:53.197927 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 02:12:53.198026 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 02:12:53.198312 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":3}
I20260929 02:12:53.198611 32111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.188546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.24.216.189:34095)
0929 02:12:53.188702 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.188707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.188722 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:12:53.188761 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.188765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.188827 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.188956 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.189537 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.190280 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.191385 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.191769 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.191894 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.194468 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.194472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.194475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.194500 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.196546 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.197079 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.197082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.197084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.197203 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.197493 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.197496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.197497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.197728 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.197825 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.198144 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.198379 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":49}
I20260929 02:12:53.203081 32108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: Bootstrap starting.
I20260929 02:12:53.205900 32108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:53.206911 32108 log.cc:824] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:53.209312 32108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: No bootstrap required, opened a new log
I20260929 02:12:53.217545 32108 raft_consensus.cc:348] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } }
I20260929 02:12:53.217948 32108 raft_consensus.cc:374] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:53.218108 32108 raft_consensus.cc:729] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c752a8f94421423dbdd6388c0a1d8845, State: Initialized, Role: FOLLOWER
I20260929 02:12:53.218920 32108 consensus_queue.cc:260] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } }
I20260929 02:12:53.219914 32114 sys_catalog.cc:455] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } } }
I20260929 02:12:53.220320 32114 sys_catalog.cc:458] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:12:53.221180 32108 sys_catalog.cc:564] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:12:53.229602 32125 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:12:53.229835 32125 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:12:53.231297 32041 master_runner.cc:428] Detected that this master c752a8f94421423dbdd6388c0a1d8845 is joining an existing cluster
I20260929 02:12:53.231389 32041 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.216.189:34095
I20260929 02:12:53.252357 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.235614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44434 (local address 127.24.216.190:39637)
0929 02:12:53.235803 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.235813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.237412 (+ 1599us) server_negotiation.cc:407] Connection header received
0929 02:12:53.237595 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.237600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.237665 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.237764 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.238893 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.239993 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.242084 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.242741 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.242838 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.245423 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.245429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.245434 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.245474 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.249136 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.249944 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.249950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.249954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.250107 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.250606 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.250610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.250615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.250995 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.251116 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.251767 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.251973 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 02:12:53.252419 32111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.235843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.24.216.189:34095)
0929 02:12:53.236056 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.236064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.236716 (+ 652us) server_negotiation.cc:407] Connection header received
0929 02:12:53.236915 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.236920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.237004 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.237122 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.238538 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.239656 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.241707 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.242564 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.242676 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.245302 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.245309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.245316 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.245351 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.249249 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.250165 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.250173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.250179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.250427 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.250895 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.250903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.250908 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.251207 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.251324 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.251716 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.252040 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":56}
I20260929 02:12:53.252835 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.235479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:34095 (local address 127.0.0.1:37456)
0929 02:12:53.236589 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:53.236617 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:53.236778 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:53.237278 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:53.237283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:53.237921 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:53.238373 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.238385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.239865 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.239871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.241536 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.241555 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.242977 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.242985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.243199 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.243851 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:53.243876 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:53.245129 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:53.249398 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.249405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.249412 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.250017 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.250580 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.250587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.250594 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.250766 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.251382 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:53.251384 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:53.251477 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 02:12:53.251628 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 02:12:53.251917 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"thread_start_us":1544,"threads_started":1}
I20260929 02:12:53.253412 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.235475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:44434)
0929 02:12:53.237317 (+ 1842us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:53.237329 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:53.237465 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:53.237778 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:53.237782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:53.238374 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:53.238747 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.238756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.240220 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.240229 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.241915 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.241928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.243127 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.243133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.243310 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.244037 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:53.244057 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:53.245230 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:53.249291 (+ 4061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.249296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.249300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.249800 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.250300 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.250304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.250306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.250469 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.251213 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:53.251218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:53.251414 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 02:12:53.252618 (+ 1204us) client_negotiation.cc:240] Negotiation successful
0929 02:12:53.252952 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1726,"mutex_wait_us":61,"spinlock_wait_cycles":32512,"thread_start_us":694,"threads_started":1}
WARNING: no policy specified for oryx/127.24.216.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.216.188@KRBTEST.COM" created.
Sep 29 02:12:53 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647971, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.190@KRBTEST.COM for oryx/127.24.216.189@KRBTEST.COM
I20260929 02:12:53.280014 32111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.259868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.24.216.189:34095)
0929 02:12:53.260040 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.260048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.262910 (+ 2862us) server_negotiation.cc:407] Connection header received
0929 02:12:53.263267 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.263273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.263371 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.263477 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.264901 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.265970 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.268040 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.268726 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.268855 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.272475 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.272484 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.272491 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.272537 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.276767 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.277620 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.277625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.277629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.277787 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.278246 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.278251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.278255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.278580 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.278768 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.279350 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.279600 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:12:53.280108 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.259836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:34095 (local address 127.0.0.1:37462)
0929 02:12:53.262699 (+ 2863us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:53.262740 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:53.263051 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:53.263550 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:53.263559 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:53.264427 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:53.264721 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.264735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.266128 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.266135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.267884 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.267898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.268892 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.268896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.269069 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.269711 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:53.269728 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:53.272282 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:53.276916 (+ 4634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.276928 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.276936 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.277479 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.277913 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.277919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.277924 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.278092 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.278747 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:53.278759 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:53.278983 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 02:12:53.279185 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 02:12:53.279566 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2628,"thread_start_us":2471,"threads_started":1}
I20260929 02:12:53.282210 31579 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.216.189:34095: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845"
member_type: NON_VOTER
last_known_addr {
host: "127.24.216.189"
port: 34095
}
attrs {
promote: true
}
}
dest_uuid: "75301924e06d4b8682d589278c402ce8"
cas_config_opid_index: -1
I20260929 02:12:53.283382 31579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: NON_VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: true } }
I20260929 02:12:53.294036 32137 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c752a8f94421423dbdd6388c0a1d8845 (127.24.216.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: NON_VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: true } } }
I20260929 02:12:53.295607 32083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:53.298230 32137 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.216.189:34095
I20260929 02:12:53.298928 32041 master.cc:537] Master@127.24.216.189:34095 shutting down...
I20260929 02:12:53.299017 32138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: NON_VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: true } } }
I20260929 02:12:53.300221 32138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:53.301033 32083 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 FOLLOWER]: Refusing update from remote peer 75301924e06d4b8682d589278c402ce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.216.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.188.keytab.
Entry for principal oryx/127.24.216.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.188.keytab.
I20260929 02:12:53.302579 32137 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: NON_VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:53.304121 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.188:43899
--webserver_interface=127.24.216.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.188.keytab
--principal=oryx/127.24.216.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:39637,127.24.216.189:34095,127.24.216.188:43899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:12:53.305517 31564 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 02:12:53.312955 32041 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:53.313377 32041 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:53.313592 32041 tablet_replica.cc:330] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: stopping tablet replica
W20260929 02:12:53.318063 31564 consensus_peers.cc:487] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 -> Peer c752a8f94421423dbdd6388c0a1d8845 (127.24.216.189:34095): Couldn't send request to peer c752a8f94421423dbdd6388c0a1d8845. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260929 02:12:53.320511 32041 master.cc:552] Master@127.24.216.189:34095 shutdown complete.
I20260929 02:12:53.320644 32041 master_runner.cc:305] Clearing existing system tablet
I20260929 02:12:53.321146 32041 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:53.321237 32041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:53.324267 32041 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260929 02:12:53.325930 32143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:53.326525 32041 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:53.326807 32041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260929 02:12:53.326934 32041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/wal
uuid: "c752a8f94421423dbdd6388c0a1d8845"
format_stamp: "Formatted at 2026-09-29 02:12:53 on dist-test-slave-9k0h"
server_key: "3b577079c651d93c4595d912ae84082f"
server_key_iv: "4a43fae91609bac8f833b2c2c35d91ea"
server_key_version: "encryptionkey@0"
I20260929 02:12:53.332069 32041 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:53.338718 32041 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 02:12:53.338907 32041 log.cc:1196] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 02:12:53.339315 32041 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: Deleting consensus metadata
I20260929 02:12:53.339607 32041 master_runner.cc:315] Copying system tablet from 127.24.216.190:39637
I20260929 02:12:53.342036 32041 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: tablet copy: Beginning tablet copy session from remote peer at address 127.24.216.190:39637
I20260929 02:12:53.357627 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.342905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44440 (local address 127.24.216.190:39637)
0929 02:12:53.343070 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.343076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.343830 (+ 754us) server_negotiation.cc:407] Connection header received
0929 02:12:53.344089 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.344095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.344194 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.344303 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.345270 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.346443 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.348260 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.348900 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.349032 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.351212 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.351218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.351222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.351257 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.354800 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.355528 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.355536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.355540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.355725 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.356155 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.356162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.356165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.356498 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.356659 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.357076 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.357294 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:12:53.357746 32149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.343211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:44440)
0929 02:12:53.343717 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:53.343736 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:53.343938 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:53.344343 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:53.344349 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:53.344878 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:53.345110 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.345121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.346605 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.346611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.348110 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.348124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.349059 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.349065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.349245 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.349946 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:53.349968 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:53.350997 (+ 1029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:53.354931 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.354936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.354941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.355400 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.355847 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.355852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.355855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.356026 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.356625 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:53.356630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:53.356812 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 02:12:53.356966 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 02:12:53.357282 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":258,"threads_started":1}
I20260929 02:12:53.358879 31599 tablet_copy_service.cc:140] P 75301924e06d4b8682d589278c402ce8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c752a8f94421423dbdd6388c0a1d8845 ({username='oryx', principal='oryx/127.24.216.189@KRBTEST.COM'} at 127.0.0.1:44440)
I20260929 02:12:53.359441 31599 tablet_copy_service.cc:161] P 75301924e06d4b8682d589278c402ce8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c752a8f94421423dbdd6388c0a1d8845 at {username='oryx', principal='oryx/127.24.216.189@KRBTEST.COM'} at 127.0.0.1:44440: session id = c752a8f94421423dbdd6388c0a1d8845-00000000000000000000000000000000
I20260929 02:12:53.364075 31599 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 02:12:53.366568 32041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 02:12:53.373174 32041 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: tablet copy: Starting download of 0 data blocks...
I20260929 02:12:53.373461 32041 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: tablet copy: Starting download of 1 WAL segments...
I20260929 02:12:53.376199 32041 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 02:12:53.380411 31599 tablet_copy_service.cc:342] P 75301924e06d4b8682d589278c402ce8: Request end of tablet copy session c752a8f94421423dbdd6388c0a1d8845-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.216.189@KRBTEST.COM'} at 127.0.0.1:44440
I20260929 02:12:53.380616 31599 tablet_copy_service.cc:433] P 75301924e06d4b8682d589278c402ce8: ending tablet copy session c752a8f94421423dbdd6388c0a1d8845-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c752a8f94421423dbdd6388c0a1d8845
I20260929 02:12:53.392376 32041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:53.395063 32150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:53.396406 32151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:53.405882 32153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:53.406386 32041 server_base.cc:1034] running on GCE node
I20260929 02:12:53.406827 32041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:53.407449 32041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:53.408663 32041 hybrid_clock.cc:648] HybridClock initialized: now 1790647973408598 us; error 35 us; skew 500 ppm
I20260929 02:12:53.408874 32041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:53 dist-test-slave-9k0h krb5kdc[31534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647973, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:53.411463 32041 init.cc:377] Logged in from keytab as oryx/127.24.216.189@KRBTEST.COM (short username oryx)
I20260929 02:12:53.412745 32041 webserver.cc:460] Webserver started at http://127.24.216.189:33285/ using document root <none> and password file <none>
I20260929 02:12:53.413097 32041 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:53.413177 32041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:53.416332 32041 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:53.418365 32160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:53.418888 32041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:53.419024 32041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/wal
uuid: "c752a8f94421423dbdd6388c0a1d8845"
format_stamp: "Formatted at 2026-09-29 02:12:53 on dist-test-slave-9k0h"
server_key: "3b577079c651d93c4595d912ae84082f"
server_key_iv: "4a43fae91609bac8f833b2c2c35d91ea"
server_key_version: "encryptionkey@0"
I20260929 02:12:53.419183 32041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:53.431885 32041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:53.435693 32041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:53.462714 32041 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.189:34095
I20260929 02:12:53.462800 32211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.189:34095 every 8 connection(s)
I20260929 02:12:53.463739 32041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-1/data/info.pb
I20260929 02:12:53.468562 32212 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:12:53.472396 32212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: Bootstrap starting.
I20260929 02:12:53.489557 32212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:12:53.490061 32212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: Bootstrap complete.
I20260929 02:12:53.491107 32212 raft_consensus.cc:348] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: NON_VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: true } }
I20260929 02:12:53.491379 32212 raft_consensus.cc:729] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c752a8f94421423dbdd6388c0a1d8845, State: Initialized, Role: LEARNER
I20260929 02:12:53.491711 32212 consensus_queue.cc:260] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: NON_VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: true } }
I20260929 02:12:53.492594 32215 sys_catalog.cc:455] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: NON_VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: true } } }
I20260929 02:12:53.492888 32212 sys_catalog.cc:564] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:53.492962 32215 sys_catalog.cc:458] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: This master's current role is: LEARNER
I20260929 02:12:53.501976 32226 catalog_manager.cc:1259] Loaded cluster ID: c451746408114b63bc62a23d8e73de81
I20260929 02:12:53.502085 32226 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: loading cluster ID for follower catalog manager: success
I20260929 02:12:53.506244 32226 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: acquiring CA information for follower catalog manager: success
I20260929 02:12:53.508616 32226 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 02:12:53.690824 32141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:53.691102 32141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:53.691205 32141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:53.701287 32141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:12:53.701503 32141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:53.701624 32141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:53.701759 32141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:12:53.701853 32141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:12:53.701900 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.694809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:34095 (local address 127.0.0.1:37478)
0929 02:12:53.695054 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:53.695078 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:53.695219 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:53.696099 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:53.696106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:53.696138 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:53.696363 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.696377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.697703 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.697710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.699839 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.699855 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.700965 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.700972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.701203 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.701270 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 02:12:53.701383 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 02:12:53.701536 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 02:12:53.701958 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.695000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.24.216.189:34095)
0929 02:12:53.695575 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.695579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.695596 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:12:53.695655 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.695658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.695725 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.695832 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:53.696543 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.697559 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.699997 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.700776 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.700995 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.701126 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.701403 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.701551 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":322,"threads_started":1}
I20260929 02:12:53.703228 32187 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 02:12:53.703830 32215 sys_catalog.cc:455] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75301924e06d4b8682d589278c402ce8. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: NON_VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: true } } }
I20260929 02:12:53.704100 32215 sys_catalog.cc:458] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: This master's current role is: LEARNER
I20260929 02:12:53.706380 32035 heartbeater.cc:503] Master 127.24.216.190:39637 was elected leader, sending a full tablet report...
I20260929 02:12:53.713560 32141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:39637,127.24.216.189:34095,127.24.216.188:43899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.188.keytab
--principal=oryx/127.24.216.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.188:43899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.216.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:53.715179 32141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:53.716832 32141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:53.729733 32232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:53.729967 32235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:53.730007 32233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:53.731298 32141 server_base.cc:1034] running on GCE node
I20260929 02:12:53.732066 32141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:53.733459 32141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:53.734659 32141 hybrid_clock.cc:648] HybridClock initialized: now 1790647973734623 us; error 29 us; skew 500 ppm
I20260929 02:12:53.734997 32141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:53 dist-test-slave-9k0h krb5kdc[31534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647973, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:53.738255 32141 init.cc:377] Logged in from keytab as oryx/127.24.216.188@KRBTEST.COM (short username oryx)
I20260929 02:12:53.739631 32141 webserver.cc:460] Webserver started at http://127.24.216.188:41867/ using document root <none> and password file <none>
I20260929 02:12:53.740329 32141 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:53.740479 32141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:53.740780 32141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:12:53.743783 32141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/data/instance:
uuid: "c0e0d77876464385970af94fe0e8c39b"
format_stamp: "Formatted at 2026-09-29 02:12:53 on dist-test-slave-9k0h"
server_key: "97843e1abee33fbaec855a69d6db5e01"
server_key_iv: "43799078971eb8ff9d13712ae5d03cc3"
server_key_version: "encryptionkey@0"
I20260929 02:12:53.744578 32141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/wal/instance:
uuid: "c0e0d77876464385970af94fe0e8c39b"
format_stamp: "Formatted at 2026-09-29 02:12:53 on dist-test-slave-9k0h"
server_key: "97843e1abee33fbaec855a69d6db5e01"
server_key_iv: "43799078971eb8ff9d13712ae5d03cc3"
server_key_version: "encryptionkey@0"
I20260929 02:12:53.749897 32141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:12:53.753443 32242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:53.754786 32141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:12:53.754967 32141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/wal
uuid: "c0e0d77876464385970af94fe0e8c39b"
format_stamp: "Formatted at 2026-09-29 02:12:53 on dist-test-slave-9k0h"
server_key: "97843e1abee33fbaec855a69d6db5e01"
server_key_iv: "43799078971eb8ff9d13712ae5d03cc3"
server_key_version: "encryptionkey@0"
I20260929 02:12:53.755162 32141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:53.767530 32141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:53.771474 32141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:53.771752 32141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:53.793566 32141 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.188:43899
I20260929 02:12:53.793586 32293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.188:43899 every 8 connection(s)
I20260929 02:12:53.795115 32141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/data/info.pb
I20260929 02:12:53.799407 32294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:53.803174 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 32141
I20260929 02:12:53.807669 32294 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:12:53 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647973, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.188@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
Sep 29 02:12:53 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.216.188@KRBTEST.COM
I20260929 02:12:53.832846 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.810672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44442 (local address 127.24.216.190:39637)
0929 02:12:53.810895 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.810901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.811292 (+ 391us) server_negotiation.cc:407] Connection header received
0929 02:12:53.811487 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.811491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.811550 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.811631 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.813346 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.814489 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.816395 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.816905 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.817040 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.825685 (+ 8645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.825690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.825694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.825729 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.829220 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.830194 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.830203 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.830207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.830341 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.830832 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.830835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.830837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.831255 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.831428 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.832321 (+ 893us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.832496 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 02:12:53.834049 32298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.810407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:44442)
0929 02:12:53.811122 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:53.811176 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:53.811364 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:53.811741 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:53.811749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:53.812645 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:53.813157 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.813176 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.814632 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.814636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.816256 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.816270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.817171 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.817178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.817369 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.823034 (+ 5665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:53.823062 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:53.825469 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:53.829359 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.829373 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.829390 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.829996 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.830464 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.830469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.830474 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.830716 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.831422 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:53.831428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:53.831886 (+ 458us) client_negotiation.cc:769] Sending connection context
0929 02:12:53.832193 (+ 307us) client_negotiation.cc:240] Negotiation successful
0929 02:12:53.832728 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":48,"thread_start_us":327,"threads_started":1}
I20260929 02:12:53.834086 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.805103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.24.216.188:43899)
0929 02:12:53.805921 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.805932 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.805974 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:12:53.808176 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.808198 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.808642 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.809051 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.810737 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.813169 (+ 2432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.815005 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.816073 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.816281 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.827200 (+ 10919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.827209 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.827224 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.827251 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.830798 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.831594 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.831601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.831604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.831749 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.832093 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.832097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.832100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.832487 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.832778 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.833544 (+ 766us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.833734 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":604,"thread_start_us":378,"threads_started":1}
I20260929 02:12:53.836200 32294 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:12:53 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647973, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.188@KRBTEST.COM for oryx/127.24.216.189@KRBTEST.COM
I20260929 02:12:53.848100 32298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.837017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:34095 (local address 127.0.0.1:37490)
0929 02:12:53.837320 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:53.837334 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:53.837462 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:53.837742 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:53.837745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:53.838028 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:53.838209 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.838226 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.839147 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.839151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.840056 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.840066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.840705 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.840709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.840850 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.841336 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:53.841351 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:53.843275 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:53.846197 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.846201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.846203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.846568 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.846893 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.846896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.846898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.847015 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.847473 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:53.847476 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:53.847578 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 02:12:53.847670 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 02:12:53.847868 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260929 02:12:53.848174 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.837016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.24.216.189:34095)
0929 02:12:53.837204 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.837210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.837425 (+ 215us) server_negotiation.cc:407] Connection header received
0929 02:12:53.837565 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.837569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.837630 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.837713 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.838333 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.839037 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.840183 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.840583 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.840677 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.843445 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.843450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.843453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.843481 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.846044 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.846661 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.846664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.846666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.846791 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.847113 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.847116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.847118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.847381 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.847485 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.847769 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.847928 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:12:53.849570 32294 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:12:53 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647973, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.188@KRBTEST.COM for oryx/127.24.216.188@KRBTEST.COM
I20260929 02:12:53.861047 32298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.850383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:43899 (local address 127.0.0.1:57116)
0929 02:12:53.850525 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:53.850551 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:53.850630 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:53.850861 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:53.850864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:53.851192 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:53.851324 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.851331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.852205 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.852214 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.853007 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.853017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.853669 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.853674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.853897 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.854429 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:53.854443 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:53.856332 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:53.858926 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.858929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.858931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.859273 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.859607 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.859611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.859615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.859825 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.860303 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:53.860308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:53.860468 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 02:12:53.860605 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 02:12:53.860780 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":4}
I20260929 02:12:53.861117 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.850313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.24.216.188:43899)
0929 02:12:53.850468 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.850472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.850607 (+ 135us) server_negotiation.cc:407] Connection header received
0929 02:12:53.850695 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.850698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.850758 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.850831 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.851450 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.852089 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.853143 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.853536 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.853663 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.856488 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.856492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.856495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.856520 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.858808 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.859373 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.859376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.859378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.859491 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.859908 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.859911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.859913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.860148 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.860343 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.860689 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.860876 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":47}
I20260929 02:12:53.865588 32294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: Bootstrap starting.
I20260929 02:12:53.868839 32294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:53.870261 32294 log.cc:824] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:53.872705 32294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: No bootstrap required, opened a new log
I20260929 02:12:53.879546 32294 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } }
I20260929 02:12:53.879849 32294 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:53.879956 32294 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e0d77876464385970af94fe0e8c39b, State: Initialized, Role: FOLLOWER
I20260929 02:12:53.880483 32294 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } }
I20260929 02:12:53.881357 32300 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } } }
I20260929 02:12:53.881692 32300 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:12:53.882371 32294 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:53.889783 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 31640
W20260929 02:12:53.891927 32311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:12:53.892167 32311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:12:53.896469 32141 master_runner.cc:428] Detected that this master c0e0d77876464385970af94fe0e8c39b is joining an existing cluster
I20260929 02:12:53.896584 32141 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.216.188:43899
Sep 29 02:12:53 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.216.129@KRBTEST.COM
I20260929 02:12:53.914050 32313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.891500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.24.216.129:38757)
0929 02:12:53.892858 (+ 1358us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.892869 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.892898 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:12:53.892967 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.892974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.893220 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.893403 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.895069 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.896345 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.897722 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.898404 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.898560 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.904682 (+ 6122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.904700 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.904721 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.904773 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.909565 (+ 4792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.910584 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.910596 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.910603 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.910807 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.911389 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.911398 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.911404 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.911970 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.912449 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.913258 (+ 809us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.913604 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1195,"thread_start_us":1037,"threads_started":1}
I20260929 02:12:53.917189 31763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:53.920102 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.902103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.24.216.189:34095)
0929 02:12:53.902319 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.902325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.902993 (+ 668us) server_negotiation.cc:407] Connection header received
0929 02:12:53.903395 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.903400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.903476 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.903546 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.905037 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.906175 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.908504 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.909573 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.909667 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.912557 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.912564 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.912572 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.912612 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.916019 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.916850 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.916857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.916861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.917028 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.917497 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.917504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.917509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.917878 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.918007 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.918910 (+ 903us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.919350 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 02:12:53.923177 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.902176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:34095 (local address 127.0.0.1:37498)
0929 02:12:53.902883 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:53.902903 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:53.903108 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:53.903751 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:53.903756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:53.904482 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:53.904862 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.904875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.906371 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.906376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.908336 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.908350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.910004 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.910012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.910395 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.911136 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:53.911158 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:53.912367 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:53.916231 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.916237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.916241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.916706 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.917170 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.917175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.917178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.917360 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.918104 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:53.918109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:53.918381 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 02:12:53.920362 (+ 1981us) client_negotiation.cc:240] Negotiation successful
0929 02:12:53.921434 (+ 1072us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":947,"threads_started":1}
I20260929 02:12:53.939045 32319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.902835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:43899 (local address 127.0.0.1:57124)
0929 02:12:53.914011 (+ 11176us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:53.914032 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:53.914403 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:53.914823 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:53.914828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:53.915491 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:53.915787 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.915799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.922294 (+ 6495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.922299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.924868 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.924887 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.926194 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.926201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.926386 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.927089 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:53.927110 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:53.928327 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:53.933203 (+ 4876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.933210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.933215 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.934735 (+ 1520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.935682 (+ 947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.935690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.935696 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.935891 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.936690 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:53.936694 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:53.936960 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 02:12:53.937108 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 02:12:53.937435 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11026,"thread_start_us":11277,"threads_started":1}
I20260929 02:12:53.939425 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.903478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.24.216.188:43899)
0929 02:12:53.903682 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.903690 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.914104 (+ 10414us) server_negotiation.cc:407] Connection header received
0929 02:12:53.914534 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.914539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.914601 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.914663 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.915946 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.916774 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.925033 (+ 8259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.925775 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.925871 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.928522 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.928529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.928535 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.928577 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.933071 (+ 4494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.934904 (+ 1833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.934912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.934918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.935527 (+ 609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.936028 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.936036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.936042 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.936555 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.936698 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.937807 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.938069 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":63}
I20260929 02:12:53.945559 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.909945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:44456)
0929 02:12:53.925208 (+ 15263us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:53.925227 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:53.925621 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:53.926502 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:53.926506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:53.927023 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:53.927226 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.927237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.929860 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.929866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.931558 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.931577 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.933922 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.933928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.936258 (+ 2330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.936958 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:53.936984 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:53.938824 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:53.942764 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.942769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.942774 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.943317 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.943723 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.943728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.943732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.943870 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.944466 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:53.944472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:53.944665 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 02:12:53.944841 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 02:12:53.945119 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15164,"mutex_wait_us":5,"thread_start_us":26295,"threads_started":1}
I20260929 02:12:53.945564 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.909354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.24.216.190:39637)
0929 02:12:53.909532 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.909537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.925317 (+ 15780us) server_negotiation.cc:407] Connection header received
0929 02:12:53.925897 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.925903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.925986 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.926057 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.928349 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.929535 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.932941 (+ 3406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.933772 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.933934 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.939016 (+ 5082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.939022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.939027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.939076 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.942620 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.943455 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.943461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.943465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.943604 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.943990 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.943995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.943999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.944340 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.944482 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.944975 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.945181 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 02:12:53 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647971, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.190@KRBTEST.COM for oryx/127.24.216.188@KRBTEST.COM
I20260929 02:12:53.968137 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.951350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:43899 (local address 127.0.0.1:57132)
0929 02:12:53.952193 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:53.952212 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:53.952335 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:53.952803 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:53.952818 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:53.953282 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:53.953503 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.953513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.954942 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.954948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.956346 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:53.956360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.957321 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.957327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:53.957497 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.958077 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:53.958100 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:53.961059 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:53.965531 (+ 4472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.965537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.965545 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.966039 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.966502 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:53.966507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:53.966511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:53.966640 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:53.967214 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:53.967220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:53.967380 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 02:12:53.967501 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 02:12:53.967763 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724}
I20260929 02:12:53.969036 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:53.951511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.24.216.188:43899)
0929 02:12:53.951662 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:53.951667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:53.952299 (+ 632us) server_negotiation.cc:407] Connection header received
0929 02:12:53.952501 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:53.952507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:53.952632 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:53.952746 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:53.953683 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.954804 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.956485 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:53.957172 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:53.957297 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:53.961255 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:53.961261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:53.961266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:53.961313 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:53.965382 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.966233 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.966238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.966242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.966378 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:53.966770 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:53.966775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:53.966778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:53.967086 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:53.968368 (+ 1282us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:53.968490 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 02:12:53.968703 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:12:53.970448 31579 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.216.188:43899: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c0e0d77876464385970af94fe0e8c39b"
member_type: NON_VOTER
last_known_addr {
host: "127.24.216.188"
port: 43899
}
attrs {
promote: true
}
}
dest_uuid: "75301924e06d4b8682d589278c402ce8"
cas_config_opid_index: 5
I20260929 02:12:53.971429 31579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: NON_VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: true } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } }
I20260929 02:12:53.973481 32138 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c0e0d77876464385970af94fe0e8c39b (127.24.216.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: NON_VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: true } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } } }
I20260929 02:12:53.975051 32187 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 LEARNER]: Refusing update from remote peer 75301924e06d4b8682d589278c402ce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 02:12:53.975312 32269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:53.981822 32138 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.216.188:43899
I20260929 02:12:53.984529 32269 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 1 FOLLOWER]: Refusing update from remote peer 75301924e06d4b8682d589278c402ce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 02:12:53.985271 32141 master.cc:537] Master@127.24.216.188:43899 shutting down...
I20260929 02:12:54.011989 32324 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: NON_VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 02:12:54.023875 32323 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:12:54.028884 32248 tablet.cc:2369] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:12:54.029881 31565 consensus_peers.cc:487] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 -> Peer c0e0d77876464385970af94fe0e8c39b (127.24.216.188:43899): Couldn't send request to peer c0e0d77876464385970af94fe0e8c39b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:12:54.030500 32187 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c0e0d77876464385970af94fe0e8c39b (127.24.216.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: NON_VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: true } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } } }
I20260929 02:12:54.033286 32141 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:54.033682 32141 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:54.033911 32141 tablet_replica.cc:330] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: stopping tablet replica
I20260929 02:12:54.034188 32324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: NON_VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: true } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } } }
I20260929 02:12:54.034611 32324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:54.037132 32215 sys_catalog.cc:455] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: NON_VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: true } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } } }
I20260929 02:12:54.037400 32215 sys_catalog.cc:458] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: This master's current role is: LEARNER
I20260929 02:12:54.038461 32138 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8: attempting to promote NON_VOTER c752a8f94421423dbdd6388c0a1d8845 to VOTER
I20260929 02:12:54.039350 32138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } }
I20260929 02:12:54.041255 32141 master.cc:552] Master@127.24.216.188:43899 shutdown complete.
I20260929 02:12:54.042105 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:54.041695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:43899 (local address 127.0.0.1:57148)
0929 02:12:54.041851 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:54.041954 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:43899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 02:12:54.042203 32186 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 LEARNER]: Refusing update from remote peer 75301924e06d4b8682d589278c402ce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 02:12:54.043074 32141 master_runner.cc:305] Clearing existing system tablet
I20260929 02:12:54.043172 32323 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 02:12:54.043489 32141 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:54.043610 32141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:54.045310 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:54.042993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:43899 (local address 127.0.0.1:57162)
0929 02:12:54.045052 (+ 2059us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:54.045152 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:43899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1956}
W20260929 02:12:54.045650 31565 consensus_peers.cc:487] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 -> Peer c0e0d77876464385970af94fe0e8c39b (127.24.216.188:43899): Couldn't send request to peer c0e0d77876464385970af94fe0e8c39b. Status: Network error: Client connection negotiation failed: client connection to 127.24.216.188:43899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 02:12:54.047230 32323 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c752a8f94421423dbdd6388c0a1d8845 (127.24.216.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } } }
I20260929 02:12:54.055845 32187 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c752a8f94421423dbdd6388c0a1d8845 (127.24.216.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } } }
I20260929 02:12:54.059036 32323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } } }
I20260929 02:12:54.059306 32323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:54.059664 32322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } } }
I20260929 02:12:54.060012 32322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:54.061786 32215 sys_catalog.cc:455] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } } }
I20260929 02:12:54.062093 32215 sys_catalog.cc:458] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:12:54.063410 32141 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.011s sys 0.007s
I20260929 02:12:54.065083 32330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:54.065526 32141 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:54.065771 32141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260929 02:12:54.065924 32141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/wal
uuid: "c0e0d77876464385970af94fe0e8c39b"
format_stamp: "Formatted at 2026-09-29 02:12:53 on dist-test-slave-9k0h"
server_key: "97843e1abee33fbaec855a69d6db5e01"
server_key_iv: "43799078971eb8ff9d13712ae5d03cc3"
server_key_version: "encryptionkey@0"
I20260929 02:12:54.070999 32141 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:54.078774 32141 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 02:12:54.078938 32141 log.cc:1196] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 02:12:54.079512 32141 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: Deleting consensus metadata
I20260929 02:12:54.079815 32141 master_runner.cc:315] Copying system tablet from 127.24.216.190:39637
I20260929 02:12:54.080543 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 31640
I20260929 02:12:54.105540 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:38757
--local_ip_for_outbound_sockets=127.24.216.129
--tserver_master_addrs=127.24.216.190:39637
--webserver_port=36301
--webserver_interface=127.24.216.129
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.129.keytab
--principal=oryx/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.216.190:39637,127.24.216.189:34095
--tserver_master_addrs=127.24.216.190:39637,127.24.216.189:34095,127.24.216.188:43899 with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:12:54.115358 32141 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: tablet copy: Beginning tablet copy session from remote peer at address 127.24.216.190:39637
I20260929 02:12:54.129796 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:54.116228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:44460)
0929 02:12:54.116861 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:54.116881 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:54.117082 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:54.117444 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:54.117448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:54.117920 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:54.118132 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:54.118141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.119564 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.119570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:54.120860 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:54.120872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.121786 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.121791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:54.121979 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:54.122782 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:54.122809 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:54.123617 (+ 808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:54.127110 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:54.127116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:54.127121 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:54.127515 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:54.127899 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:54.127904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:54.127908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:54.128053 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:54.128585 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:54.128590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:54.128790 (+ 200us) client_negotiation.cc:769] Sending connection context
0929 02:12:54.128942 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 02:12:54.129330 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":359,"threads_started":1}
I20260929 02:12:54.129850 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:54.116235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.24.216.190:39637)
0929 02:12:54.116453 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:54.116462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:54.116984 (+ 522us) server_negotiation.cc:407] Connection header received
0929 02:12:54.117218 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:54.117224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:54.117308 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:54.117417 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:54.118399 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.119409 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.121007 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.121599 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.121778 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:54.123811 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:54.123818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:54.123825 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:54.123861 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:54.126979 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:54.127635 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:54.127641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:54.127645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:54.127781 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:54.128163 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:54.128168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:54.128172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:54.128461 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:54.128621 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:54.129256 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 02:12:54.129487 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 02:12:54.130882 31599 tablet_copy_service.cc:140] P 75301924e06d4b8682d589278c402ce8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c0e0d77876464385970af94fe0e8c39b ({username='oryx', principal='oryx/127.24.216.188@KRBTEST.COM'} at 127.0.0.1:44460)
I20260929 02:12:54.131063 31599 tablet_copy_service.cc:161] P 75301924e06d4b8682d589278c402ce8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c0e0d77876464385970af94fe0e8c39b at {username='oryx', principal='oryx/127.24.216.188@KRBTEST.COM'} at 127.0.0.1:44460: session id = c0e0d77876464385970af94fe0e8c39b-00000000000000000000000000000000
I20260929 02:12:54.133337 31599 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 02:12:54.135030 32141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 02:12:54.139914 32141 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: tablet copy: Starting download of 0 data blocks...
I20260929 02:12:54.140218 32141 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: tablet copy: Starting download of 1 WAL segments...
I20260929 02:12:54.142426 32141 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 02:12:54.145569 31599 tablet_copy_service.cc:342] P 75301924e06d4b8682d589278c402ce8: Request end of tablet copy session c0e0d77876464385970af94fe0e8c39b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.216.188@KRBTEST.COM'} at 127.0.0.1:44460
I20260929 02:12:54.145706 31599 tablet_copy_service.cc:433] P 75301924e06d4b8682d589278c402ce8: ending tablet copy session c0e0d77876464385970af94fe0e8c39b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c0e0d77876464385970af94fe0e8c39b
I20260929 02:12:54.158105 32141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:54.160909 32338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:54.167325 32339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:54.174391 32141 server_base.cc:1034] running on GCE node
W20260929 02:12:54.174969 32341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:54.175424 32141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:54.176090 32141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:54.177332 32141 hybrid_clock.cc:648] HybridClock initialized: now 1790647974177280 us; error 39 us; skew 500 ppm
I20260929 02:12:54.177597 32141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:54 dist-test-slave-9k0h krb5kdc[31534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647974, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:54.180030 32141 init.cc:377] Logged in from keytab as oryx/127.24.216.188@KRBTEST.COM (short username oryx)
I20260929 02:12:54.181406 32141 webserver.cc:460] Webserver started at http://127.24.216.188:42027/ using document root <none> and password file <none>
I20260929 02:12:54.181785 32141 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:54.181893 32141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:54.185003 32141 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:54.189230 32348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:54.189662 32141 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260929 02:12:54.189790 32141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/wal
uuid: "c0e0d77876464385970af94fe0e8c39b"
format_stamp: "Formatted at 2026-09-29 02:12:53 on dist-test-slave-9k0h"
server_key: "97843e1abee33fbaec855a69d6db5e01"
server_key_iv: "43799078971eb8ff9d13712ae5d03cc3"
server_key_version: "encryptionkey@0"
I20260929 02:12:54.190002 32141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:54.211587 32141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:54.216904 32141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:54.264855 32141 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.188:43899
I20260929 02:12:54.264909 32399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.188:43899 every 8 connection(s)
I20260929 02:12:54.265926 32141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/master-2/data/info.pb
I20260929 02:12:54.271762 32400 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:12:54.275942 32400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: Bootstrap starting.
I20260929 02:12:54.294971 32400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:12:54.295509 32400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: Bootstrap complete.
I20260929 02:12:54.296685 32400 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } }
I20260929 02:12:54.297060 32400 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c0e0d77876464385970af94fe0e8c39b, State: Initialized, Role: LEARNER
I20260929 02:12:54.297364 32400 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } }
I20260929 02:12:54.298659 32400 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:12:54.298359 32403 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } } }
I20260929 02:12:54.298997 32403 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [sys.catalog]: This master's current role is: LEARNER
I20260929 02:12:54.313977 32414 catalog_manager.cc:1259] Loaded cluster ID: c451746408114b63bc62a23d8e73de81
I20260929 02:12:54.314091 32414 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: loading cluster ID for follower catalog manager: success
I20260929 02:12:54.317621 32414 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: acquiring CA information for follower catalog manager: success
I20260929 02:12:54.320304 32414 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 02:12:54.506780 32333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:54.507033 32333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:54.507305 32333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:54.514707 32333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:54.514827 32333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:54.515000 32333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:12:54.524616 32333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.129.keytab
--principal=oryx/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:38757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=36301
--tserver_master_addrs=127.24.216.190:39637,127.24.216.189:34095,127.24.216.188:43899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:54.525941 32333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:54.527225 32333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:54.540614 32419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:54.540572 32422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:54.540645 32420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:54.541393 32333 server_base.cc:1034] running on GCE node
I20260929 02:12:54.542011 32333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:54.543093 32333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:54.544305 32333 hybrid_clock.cc:648] HybridClock initialized: now 1790647974544251 us; error 51 us; skew 500 ppm
I20260929 02:12:54.544602 32333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:54 dist-test-slave-9k0h krb5kdc[31534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647974, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:54.548043 32333 init.cc:377] Logged in from keytab as oryx/127.24.216.129@KRBTEST.COM (short username oryx)
I20260929 02:12:54.549467 32333 webserver.cc:460] Webserver started at http://127.24.216.129:36301/ using document root <none> and password file <none>
I20260929 02:12:54.550066 32333 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:54.550232 32333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:54.556154 32333 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:12:54.559231 32431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:54.559553 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:54.553330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.24.216.188:43899)
0929 02:12:54.553906 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:54.553913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:54.553936 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:12:54.554006 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:54.554011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:54.554132 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:54.554294 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:54.554579 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.555891 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.557434 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.558563 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.558744 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:54.558991 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:54.559128 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 02:12:54.559230 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":317,"threads_started":1}
I20260929 02:12:54.559567 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:54.553169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:43899 (local address 127.0.0.1:57166)
0929 02:12:54.553661 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:54.553683 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:54.553815 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:54.554308 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:54.554311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:54.554331 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:54.554464 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:54.554471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.556021 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.556025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:54.557294 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:54.557307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.558734 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.558743 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:54.558967 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:54.559004 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:12:54.559104 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:12:54.559251 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":217,"threads_started":1}
I20260929 02:12:54.560506 32375 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 02:12:54.561039 32333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:54.561025 32403 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75301924e06d4b8682d589278c402ce8. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } } }
I20260929 02:12:54.561309 32403 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [sys.catalog]: This master's current role is: LEARNER
I20260929 02:12:54.561184 32333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "1762517f34e643f5a4d2ad8adf685336"
format_stamp: "Formatted at 2026-09-29 02:12:51 on dist-test-slave-9k0h"
server_key: "464dd62bdc29cf491642e1e9defff8cf"
server_key_iv: "433237ab8629721348363cd8a2536abf"
server_key_version: "encryptionkey@0"
I20260929 02:12:54.562477 32333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:54.580106 32333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:54.583848 32333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:54.584074 32333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:54.584883 32333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:54.586067 32333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:54.586190 32333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:54.586287 32333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:54.586344 32333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:54.622324 32333 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:38757
I20260929 02:12:54.622344 32543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:38757 every 8 connection(s)
I20260929 02:12:54.623708 32333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:12:54.624408 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 32333
I20260929 02:12:54.624754 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 31774
Sep 29 02:12:54 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647974, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
Sep 29 02:12:54 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647974, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.189@KRBTEST.COM
Sep 29 02:12:54 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647974, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.188@KRBTEST.COM
Sep 29 02:12:54 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.216.130@KRBTEST.COM
I20260929 02:12:54.694135 32554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:54.633110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:33015 (local address 127.24.216.190:39637)
0929 02:12:54.662277 (+ 29167us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:54.662286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:54.666717 (+ 4431us) server_negotiation.cc:407] Connection header received
0929 02:12:54.667865 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:54.667873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:54.667961 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:54.668088 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:54.671310 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.672708 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.677043 (+ 4335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.678102 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.678203 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:54.685829 (+ 7626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:54.685836 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:54.685840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:54.685880 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:54.689547 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:54.690879 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:54.690884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:54.690888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:54.691021 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:54.691848 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:54.691854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:54.691857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:54.692174 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:54.692299 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:54.693446 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0929 02:12:54.693669 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28992,"thread_start_us":30144,"threads_started":1}
I20260929 02:12:54.696511 32552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:54.632621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.129:33015)
0929 02:12:54.666571 (+ 33950us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:54.666592 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:54.667679 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:54.668617 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:54.668633 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:54.669818 (+ 1185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:54.671100 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:54.671127 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.672941 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.672949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:54.676120 (+ 3171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:54.676139 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.680898 (+ 4759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.680905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:54.681176 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:54.681990 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:54.682033 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:54.685587 (+ 3554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:54.689703 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:54.689714 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:54.689732 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:54.690731 (+ 999us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:54.691169 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:54.691175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:54.691180 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:54.691696 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:54.692389 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:54.692397 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:54.693004 (+ 607us) client_negotiation.cc:769] Sending connection context
0929 02:12:54.694371 (+ 1367us) client_negotiation.cc:240] Negotiation successful
0929 02:12:54.694929 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33786,"thread_start_us":29698,"threads_started":1}
I20260929 02:12:54.696702 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:54.628401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:51005 (local address 127.24.216.189:34095)
0929 02:12:54.678461 (+ 50060us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:54.678466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:54.678488 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:12:54.678546 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:54.678550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:54.678636 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:54.678809 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:54.680500 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.681638 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.684227 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.684873 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.684989 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:54.688863 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:54.688869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:54.688874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:54.688910 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:54.693019 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:54.693907 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:54.693912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:54.693916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:54.694045 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:54.694611 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:54.694616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:54.694619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:54.694951 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:54.695971 (+ 1020us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:54.696116 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 02:12:54.696332 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49942,"thread_start_us":51422,"threads_started":1}
I20260929 02:12:54.697507 32550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:54.628399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:34095 (local address 127.24.216.129:51005)
0929 02:12:54.665886 (+ 37487us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:54.665909 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:54.667011 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:54.678813 (+ 11802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:54.678817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:54.680007 (+ 1190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:54.680353 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:54.680364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.681816 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.681824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:54.684060 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:54.684075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.685201 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.685206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:54.685378 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:54.685986 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:54.686005 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:54.688635 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:54.693161 (+ 4526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:54.693165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:54.693168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:54.693706 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:54.694206 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:54.694211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:54.694214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:54.694458 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:54.695092 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:54.695097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:54.695227 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 02:12:54.695351 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 02:12:54.695703 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37356,"mutex_wait_us":82,"thread_start_us":37813,"threads_started":1}
I20260929 02:12:54.698297 32545 heartbeater.cc:347] Connected to a master server at 127.24.216.190:39637
I20260929 02:12:54.698632 32545 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:54.699004 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:54.626758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:51223 (local address 127.24.216.188:43899)
0929 02:12:54.626943 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:54.626949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:54.661091 (+ 34142us) server_negotiation.cc:407] Connection header received
0929 02:12:54.667305 (+ 6214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:54.667311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:54.667399 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:54.667506 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:54.680746 (+ 13240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.681767 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.685226 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.685829 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.685915 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:54.691336 (+ 5421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:54.691343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:54.691347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:54.691384 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:54.695372 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:54.696500 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:54.696505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:54.696509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:54.696661 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:54.697287 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:54.697292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:54.697295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:54.697644 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:54.697777 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:54.698364 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 02:12:54.698634 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:12:54.699612 32545 heartbeater.cc:511] Master 127.24.216.190:39637 requested a full tablet report, sending...
I20260929 02:12:54.700004 32553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:54.627547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:43899 (local address 127.24.216.129:51223)
0929 02:12:54.660864 (+ 33317us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:54.660931 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:54.667011 (+ 6080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:54.679725 (+ 12714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:54.679733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:54.680241 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:54.680563 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:54.680576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.681977 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.681984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:54.685054 (+ 3070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:54.685069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.686211 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.686217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:54.686630 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:54.687219 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:54.687243 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:54.691117 (+ 3874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:54.695832 (+ 4715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:54.695837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:54.695843 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:54.696360 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:54.696918 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:54.696926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:54.696931 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:54.697114 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:54.697844 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:54.697847 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:54.698005 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 02:12:54.699163 (+ 1158us) client_negotiation.cc:240] Negotiation successful
0929 02:12:54.699553 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32854,"mutex_wait_us":145,"thread_start_us":37951,"threads_started":1}
I20260929 02:12:54.700836 32546 heartbeater.cc:347] Connected to a master server at 127.24.216.189:34095
I20260929 02:12:54.700975 32546 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:54.701326 32546 heartbeater.cc:511] Master 127.24.216.189:34095 requested a full tablet report, sending...
I20260929 02:12:54.701414 31579 ts_manager.cc:194] Re-registered known tserver with Master: 1762517f34e643f5a4d2ad8adf685336 (127.24.216.129:38757)
I20260929 02:12:54.701804 32544 heartbeater.cc:347] Connected to a master server at 127.24.216.188:43899
I20260929 02:12:54.701925 32544 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:54.702283 32544 heartbeater.cc:511] Master 127.24.216.188:43899 requested a full tablet report, sending...
I20260929 02:12:54.702468 31579 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:33015
I20260929 02:12:54.702672 32548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:54.625817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54132 (local address 127.24.216.130:36219)
0929 02:12:54.676936 (+ 51119us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:54.676946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:54.676975 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:12:54.677047 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:54.677054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:54.677282 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:54.677967 (+ 685us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:54.681885 (+ 3918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.683828 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.685517 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:54.686673 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:54.686853 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:54.692930 (+ 6077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:54.692945 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:54.692962 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:54.693011 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:54.698903 (+ 5892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:54.699802 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:54.699810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:54.699813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:54.699960 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:54.700331 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:54.700336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:54.700340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:54.700753 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:54.701882 (+ 1129us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:54.702028 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 02:12:54.702304 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50947,"thread_start_us":41086,"threads_started":1}
I20260929 02:12:54.703514 32177 ts_manager.cc:194] Registered new tserver with Master: 1762517f34e643f5a4d2ad8adf685336 (127.24.216.129:38757)
I20260929 02:12:54.704282 31897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:54.706017 32365 ts_manager.cc:194] Registered new tserver with Master: 1762517f34e643f5a4d2ad8adf685336 (127.24.216.129:38757)
I20260929 02:12:54.786412 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 31774
I20260929 02:12:54.803145 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:36219
--local_ip_for_outbound_sockets=127.24.216.130
--tserver_master_addrs=127.24.216.190:39637
--webserver_port=36047
--webserver_interface=127.24.216.130
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.130.keytab
--principal=oryx/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.216.190:39637,127.24.216.189:34095
--tserver_master_addrs=127.24.216.190:39637,127.24.216.189:34095,127.24.216.188:43899 with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:12:55.021723 32328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: NON_VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: true } } }
I20260929 02:12:55.022152 32328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:55.023526 32557 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8: attempting to promote NON_VOTER c0e0d77876464385970af94fe0e8c39b to VOTER
I20260929 02:12:55.024060 32557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } }
I20260929 02:12:55.025753 32375 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 1 LEARNER]: Refusing update from remote peer 75301924e06d4b8682d589278c402ce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 02:12:55.025810 32187 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 FOLLOWER]: Refusing update from remote peer 75301924e06d4b8682d589278c402ce8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 02:12:55.026320 32557 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 02:12:55.026677 32558 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 02:12:55.028925 32557 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c0e0d77876464385970af94fe0e8c39b (127.24.216.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } } }
I20260929 02:12:55.029489 32187 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c0e0d77876464385970af94fe0e8c39b (127.24.216.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } } }
I20260929 02:12:55.032847 32557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } } }
I20260929 02:12:55.033099 32557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:55.033344 32328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } } }
I20260929 02:12:55.033615 32328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:55.035749 32328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } } }
I20260929 02:12:55.035948 32328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:55.035449 32561 sys_catalog.cc:455] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } } }
I20260929 02:12:55.036307 32561 sys_catalog.cc:458] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:12:55.036581 32375 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c0e0d77876464385970af94fe0e8c39b (127.24.216.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } } }
I20260929 02:12:55.039443 32403 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75301924e06d4b8682d589278c402ce8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } } }
I20260929 02:12:55.039708 32403 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [sys.catalog]: This master's current role is: FOLLOWER
W20260929 02:12:55.121038 32556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:55.121330 32556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:55.121423 32556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:55.128561 32556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:55.128651 32556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:55.128755 32556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:12:55.137943 32556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.130.keytab
--principal=oryx/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:36219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=36047
--tserver_master_addrs=127.24.216.190:39637,127.24.216.189:34095,127.24.216.188:43899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:55.139204 32556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:55.140439 32556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:55.152904 32566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:55.153136 32567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:55.152994 32569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:55.154124 32556 server_base.cc:1034] running on GCE node
I20260929 02:12:55.154690 32556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:55.155560 32556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:55.156795 32556 hybrid_clock.cc:648] HybridClock initialized: now 1790647975156753 us; error 32 us; skew 500 ppm
I20260929 02:12:55.157130 32556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:55 dist-test-slave-9k0h krb5kdc[31534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647975, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:55.160244 32556 init.cc:377] Logged in from keytab as oryx/127.24.216.130@KRBTEST.COM (short username oryx)
I20260929 02:12:55.161525 32556 webserver.cc:460] Webserver started at http://127.24.216.130:36047/ using document root <none> and password file <none>
I20260929 02:12:55.162141 32556 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:55.162331 32556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:55.167574 32556 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:12:55.170055 32576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:55.171397 32556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:12:55.171558 32556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "45cffc9111ec424881246da5734a8573"
format_stamp: "Formatted at 2026-09-29 02:12:52 on dist-test-slave-9k0h"
server_key: "bf13744c295af70bf9390c572d10ea74"
server_key_iv: "9c2686c32b71e6e6ae2373daadc581cf"
server_key_version: "encryptionkey@0"
I20260929 02:12:55.172501 32556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:55.191629 32556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:55.195364 32556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:55.195698 32556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:55.196587 32556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:55.197777 32556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:55.197860 32556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:55.197983 32556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:55.198079 32556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:55.235101 32556 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:36219
I20260929 02:12:55.235157 32688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:36219 every 8 connection(s)
I20260929 02:12:55.236531 32556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:12:55.246460 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 32556
I20260929 02:12:55.246754 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 31907
Sep 29 02:12:55 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647975, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.189@KRBTEST.COM
Sep 29 02:12:55 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647975, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.188@KRBTEST.COM
Sep 29 02:12:55 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647975, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
I20260929 02:12:55.271056 32696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.240860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:46701 (local address 127.24.216.189:34095)
0929 02:12:55.241270 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:55.241275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:55.241296 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:12:55.244609 (+ 3313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:55.244615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:55.244726 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:55.244824 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:55.247579 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.249095 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.252212 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.253043 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.253158 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.259252 (+ 6094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:55.259263 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:55.259271 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:55.259349 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:55.264355 (+ 5006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.267065 (+ 2710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.267074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.267080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.267254 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.267915 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.267922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.267928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.268812 (+ 884us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:55.268996 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:55.270193 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0929 02:12:55.270501 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":620,"threads_started":1}
I20260929 02:12:55.274178 32694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.240067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:34095 (local address 127.24.216.130:46701)
0929 02:12:55.241065 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:55.241116 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:55.244295 (+ 3179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:55.244994 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:55.244997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:55.245937 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:55.247194 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.247222 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.249376 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.249385 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.252027 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.252044 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.253569 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.253578 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.254108 (+ 530us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.255059 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:55.255104 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:55.258892 (+ 3788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:55.265963 (+ 7071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:55.265979 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:55.266002 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:55.266872 (+ 870us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:55.267445 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:55.267453 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:55.267459 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:55.267767 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:55.269134 (+ 1367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:55.269146 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:55.269652 (+ 506us) client_negotiation.cc:769] Sending connection context
0929 02:12:55.271682 (+ 2030us) client_negotiation.cc:240] Negotiation successful
0929 02:12:55.272367 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":145,"thread_start_us":1395,"threads_started":1}
I20260929 02:12:55.276602 32691 heartbeater.cc:347] Connected to a master server at 127.24.216.189:34095
I20260929 02:12:55.276953 32691 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:55.277865 32691 heartbeater.cc:511] Master 127.24.216.189:34095 requested a full tablet report, sending...
I20260929 02:12:55.278705 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.241291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:43899 (local address 127.24.216.130:47097)
0929 02:12:55.243426 (+ 2135us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:55.243440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:55.244291 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:55.244979 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:55.244990 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:55.245936 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:55.251169 (+ 5233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.251182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.252975 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.252983 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.256053 (+ 3070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.256072 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.259203 (+ 3131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.259212 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.259476 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.263234 (+ 3758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:55.263265 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:55.267284 (+ 4019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:55.275202 (+ 7918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:55.275208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:55.275211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:55.275732 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:55.276824 (+ 1092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:55.276828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:55.276831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:55.276973 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:55.277666 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:55.277672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:55.277821 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 02:12:55.277968 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 02:12:55.278264 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2042,"mutex_wait_us":214,"spinlock_wait_cycles":14720,"thread_start_us":461,"threads_started":1}
I20260929 02:12:55.279400 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.239306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:47097 (local address 127.24.216.188:43899)
0929 02:12:55.242901 (+ 3595us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:55.242909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:55.243527 (+ 618us) server_negotiation.cc:407] Connection header received
0929 02:12:55.244524 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:55.244529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:55.244622 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:55.244727 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:55.251331 (+ 6604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.252766 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.257815 (+ 5049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.258710 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.258836 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.267463 (+ 8627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:55.267469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:55.267474 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:55.267509 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:55.275054 (+ 7545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.276535 (+ 1481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.276541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.276548 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.276685 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.277111 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.277116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.277118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.277452 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:55.277592 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:55.278270 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 02:12:55.278582 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3439,"thread_start_us":3308,"threads_started":1}
I20260929 02:12:55.279743 32177 ts_manager.cc:194] Registered new tserver with Master: 45cffc9111ec424881246da5734a8573 (127.24.216.130:36219)
I20260929 02:12:55.280792 32689 heartbeater.cc:347] Connected to a master server at 127.24.216.188:43899
I20260929 02:12:55.280946 32689 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:55.281417 32689 heartbeater.cc:511] Master 127.24.216.188:43899 requested a full tablet report, sending...
I20260929 02:12:55.281600 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.240067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.130:46603)
0929 02:12:55.243895 (+ 3828us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:55.243907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:55.244295 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:55.245317 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:55.245320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:55.245935 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:55.247198 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.247221 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.250569 (+ 3348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.250578 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.257615 (+ 7037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.257633 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.261399 (+ 3766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.261407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.261638 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.262258 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:55.262300 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:55.267118 (+ 4818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:55.275860 (+ 8742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:55.275864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:55.275868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:55.276383 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:55.278916 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:55.278924 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:55.278929 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:55.279083 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:55.280469 (+ 1386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:55.280474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:55.280758 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 02:12:55.280890 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 02:12:55.281166 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3707,"mutex_wait_us":1663,"spinlock_wait_cycles":20992,"thread_start_us":2097,"threads_started":1}
I20260929 02:12:55.281829 32695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.240809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:46603 (local address 127.24.216.190:39637)
0929 02:12:55.243251 (+ 2442us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:55.243255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:55.243990 (+ 735us) server_negotiation.cc:407] Connection header received
0929 02:12:55.244647 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:55.244655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:55.244735 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:55.244839 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:55.247366 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.248763 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.260039 (+ 11276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.260937 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.261058 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.268459 (+ 7401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:55.268465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:55.268470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:55.268506 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:55.272306 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.276555 (+ 4249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.276564 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.276571 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.276850 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.279598 (+ 2748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.279604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.279607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.279924 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:55.280069 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:55.281082 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0929 02:12:55.281328 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2354,"thread_start_us":2266,"threads_started":1}
I20260929 02:12:55.282619 32690 heartbeater.cc:347] Connected to a master server at 127.24.216.190:39637
I20260929 02:12:55.282730 32690 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:55.282774 32365 ts_manager.cc:194] Registered new tserver with Master: 45cffc9111ec424881246da5734a8573 (127.24.216.130:36219)
I20260929 02:12:55.283651 32690 heartbeater.cc:511] Master 127.24.216.190:39637 requested a full tablet report, sending...
I20260929 02:12:55.284475 31579 ts_manager.cc:194] Re-registered known tserver with Master: 45cffc9111ec424881246da5734a8573 (127.24.216.130:36219)
I20260929 02:12:55.285631 31579 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:46603
Sep 29 02:12:55 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.216.131@KRBTEST.COM
I20260929 02:12:55.299669 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.247637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.24.216.131:39063)
0929 02:12:55.279604 (+ 31967us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:55.279612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:55.279640 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:12:55.279705 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:55.279712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:55.279958 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:55.283499 (+ 3541us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:55.284813 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.287878 (+ 3065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.289172 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.289744 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.289845 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.294621 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:55.294630 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:55.294643 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:55.294673 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:55.297287 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.297826 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.297832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.297834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.297949 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.298278 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.298281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.298283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.298567 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:55.298803 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:55.299112 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 02:12:55.299320 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31793,"thread_start_us":31587,"threads_started":1}
I20260929 02:12:55.300956 32030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:55.383708 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 31907
I20260929 02:12:55.400609 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:39063
--local_ip_for_outbound_sockets=127.24.216.131
--tserver_master_addrs=127.24.216.190:39637
--webserver_port=45517
--webserver_interface=127.24.216.131
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.131.keytab
--principal=oryx/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.216.190:39637,127.24.216.189:34095
--tserver_master_addrs=127.24.216.190:39637,127.24.216.189:34095,127.24.216.188:43899 with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:12:55.705753 32545 heartbeater.cc:503] Master 127.24.216.190:39637 was elected leader, sending a full tablet report...
W20260929 02:12:55.717398 32702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:12:55.717643 32702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:12:55.717728 32702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:12:55.725035 32702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:12:55.725127 32702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:12:55.725255 32702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:12:55.734464 32702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.131.keytab
--principal=oryx/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:39063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=45517
--tserver_master_addrs=127.24.216.190:39637,127.24.216.189:34095,127.24.216.188:43899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:12:55.735683 32702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:12:55.736898 32702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:12:55.749521 32710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:55.749558 32708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:12:55.749594 32707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:12:55.750487 32702 server_base.cc:1034] running on GCE node
I20260929 02:12:55.751048 32702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:12:55.751855 32702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:12:55.753065 32702 hybrid_clock.cc:648] HybridClock initialized: now 1790647975753026 us; error 55 us; skew 500 ppm
I20260929 02:12:55.753352 32702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:12:55 dist-test-slave-9k0h krb5kdc[31534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647975, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:12:55.756230 32702 init.cc:377] Logged in from keytab as oryx/127.24.216.131@KRBTEST.COM (short username oryx)
I20260929 02:12:55.757520 32702 webserver.cc:460] Webserver started at http://127.24.216.131:45517/ using document root <none> and password file <none>
I20260929 02:12:55.758235 32702 fs_manager.cc:359] Metadata directory not provided
I20260929 02:12:55.758347 32702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:12:55.763252 32702 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 02:12:55.765622 32717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:55.767023 32702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:12:55.767149 32702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "979224aefeeb448287acefed22a220c9"
format_stamp: "Formatted at 2026-09-29 02:12:52 on dist-test-slave-9k0h"
server_key: "c483cd419fb98047fb98982756f358b5"
server_key_iv: "e699d5685699a70dd213fab4b1349986"
server_key_version: "encryptionkey@0"
I20260929 02:12:55.768081 32702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:12:55.782140 32702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:12:55.785889 32702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:12:55.786103 32702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:12:55.786952 32702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:12:55.788136 32702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:12:55.788216 32702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:55.788291 32702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:12:55.788357 32702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:12:55.819327 32702 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:39063
I20260929 02:12:55.819393 361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:39063 every 8 connection(s)
I20260929 02:12:55.820848 32702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:12:55.830634 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 32702
Sep 29 02:12:55 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647975, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
Sep 29 02:12:55 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647975, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.188@KRBTEST.COM
Sep 29 02:12:55 dist-test-slave-9k0h krb5kdc[31534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647975, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.189@KRBTEST.COM
I20260929 02:12:55.891863 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.823601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:47547 (local address 127.24.216.190:39637)
0929 02:12:55.843187 (+ 19586us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:55.843196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:55.843220 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:12:55.843292 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:55.843299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:55.843395 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:55.843664 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:55.846427 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.847840 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.850900 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.851739 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.851875 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.857529 (+ 5654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:55.857537 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:55.857542 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:55.857581 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:55.873897 (+ 16316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.882718 (+ 8821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.882727 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.882733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.882925 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.887876 (+ 4951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.887884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.887889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.888336 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:55.888505 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:55.890962 (+ 2457us) server_negotiation.cc:299] Negotiation successful
0929 02:12:55.891296 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19428,"thread_start_us":22653,"threads_started":1}
I20260929 02:12:55.893787 371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.824449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.131:47547)
0929 02:12:55.833822 (+ 9373us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:55.833880 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:55.838021 (+ 4141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:55.844171 (+ 6150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:55.844188 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:55.845108 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:55.846080 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.846108 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.848044 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.848052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.850688 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.850707 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.852211 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.852220 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.852583 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.853462 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:55.853498 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:55.857257 (+ 3759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:55.881715 (+ 24458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:55.881728 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:55.881753 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:55.882523 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:55.887480 (+ 4957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:55.887492 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:55.887499 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:55.887716 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:55.889387 (+ 1671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:55.889400 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:55.890633 (+ 1233us) client_negotiation.cc:769] Sending connection context
0929 02:12:55.890812 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 02:12:55.891629 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8995,"mutex_wait_us":952,"thread_start_us":31404,"threads_started":1}
I20260929 02:12:55.897375 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.824449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:34095 (local address 127.24.216.131:55461)
0929 02:12:55.856299 (+ 31850us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:55.856321 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:55.856423 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:55.856974 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:55.856980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:55.857794 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:55.858316 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.858327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.859701 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.859709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.863728 (+ 4019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.863744 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.866180 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.866189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.866427 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.869432 (+ 3005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:55.869461 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:55.877661 (+ 8200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:55.887236 (+ 9575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:55.887243 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:55.887248 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:55.887909 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:55.888410 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:55.888418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:55.888424 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:55.888597 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:55.889427 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:55.889435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:55.889999 (+ 564us) client_negotiation.cc:769] Sending connection context
0929 02:12:55.896395 (+ 6396us) client_negotiation.cc:240] Negotiation successful
0929 02:12:55.896777 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31681,"mutex_wait_us":2252,"thread_start_us":6570,"threads_started":1}
I20260929 02:12:55.899562 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.824922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:46529 (local address 127.24.216.188:43899)
0929 02:12:55.856012 (+ 31090us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:55.856019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:55.856042 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:12:55.856103 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:55.856109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:55.856194 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:55.856301 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:55.858311 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.859492 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.867017 (+ 7525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.867909 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.868023 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.880153 (+ 12130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:55.880162 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:55.880169 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:55.880220 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:55.884499 (+ 4279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.886445 (+ 1946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.886453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.886459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.886628 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.894664 (+ 8036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.894673 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.894679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.895090 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:55.895264 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:55.898661 (+ 3397us) server_negotiation.cc:299] Negotiation successful
0929 02:12:55.898954 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30962,"thread_start_us":30826,"threads_started":1}
I20260929 02:12:55.900781 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.825285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:55461 (local address 127.24.216.189:34095)
0929 02:12:55.855631 (+ 30346us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:55.855638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:55.856529 (+ 891us) server_negotiation.cc:407] Connection header received
0929 02:12:55.856595 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:55.856601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:55.856684 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:55.856790 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:55.858478 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.859539 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.863909 (+ 4370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.865019 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.865165 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.878136 (+ 12971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:55.878146 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:55.878154 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:55.878240 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:55.887083 (+ 8843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.888074 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.888082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.888088 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.888251 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.888734 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.888742 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.888747 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.889175 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:55.889355 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:55.897877 (+ 8522us) server_negotiation.cc:299] Negotiation successful
0929 02:12:55.900397 (+ 2520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30190,"thread_start_us":30078,"threads_started":1}
I20260929 02:12:55.898185 363 heartbeater.cc:347] Connected to a master server at 127.24.216.190:39637
I20260929 02:12:55.901255 363 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:55.900789 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.825630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:43899 (local address 127.24.216.131:46529)
0929 02:12:55.855001 (+ 29371us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:55.855037 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:55.855168 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:55.856491 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:55.856496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:55.857737 (+ 1241us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:12:55.858100 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.858114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.859675 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.859681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.862985 (+ 3304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.863005 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.868384 (+ 5379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.868392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.868609 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.870490 (+ 1881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:12:55.870521 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:12:55.879615 (+ 9094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:12:55.884690 (+ 5075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:55.884698 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:55.884704 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:55.885470 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:55.886811 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:12:55.886818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:12:55.886824 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:12:55.887109 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:12:55.895392 (+ 8283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:12:55.895400 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:12:55.895576 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 02:12:55.900121 (+ 4545us) client_negotiation.cc:240] Negotiation successful
0929 02:12:55.900397 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29194,"thread_start_us":28994,"threads_started":1}
I20260929 02:12:55.902104 363 heartbeater.cc:511] Master 127.24.216.190:39637 requested a full tablet report, sending...
I20260929 02:12:55.902360 364 heartbeater.cc:347] Connected to a master server at 127.24.216.189:34095
I20260929 02:12:55.902474 364 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:55.903007 364 heartbeater.cc:511] Master 127.24.216.189:34095 requested a full tablet report, sending...
I20260929 02:12:55.903904 31579 ts_manager.cc:194] Re-registered known tserver with Master: 979224aefeeb448287acefed22a220c9 (127.24.216.131:39063)
I20260929 02:12:55.904174 32177 ts_manager.cc:194] Registered new tserver with Master: 979224aefeeb448287acefed22a220c9 (127.24.216.131:39063)
I20260929 02:12:55.905094 31579 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:47547
I20260929 02:12:55.905553 362 heartbeater.cc:347] Connected to a master server at 127.24.216.188:43899
I20260929 02:12:55.905681 362 heartbeater.cc:464] Registering TS with master...
I20260929 02:12:55.906116 362 heartbeater.cc:511] Master 127.24.216.188:43899 requested a full tablet report, sending...
I20260929 02:12:55.907613 32365 ts_manager.cc:194] Registered new tserver with Master: 979224aefeeb448287acefed22a220c9 (127.24.216.131:39063)
I20260929 02:12:55.914690 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.860366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44464 (local address 127.24.216.190:39637)
0929 02:12:55.892214 (+ 31848us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:55.892220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:55.894420 (+ 2200us) server_negotiation.cc:407] Connection header received
0929 02:12:55.894620 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:55.894628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:55.894731 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:55.894821 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:55.899423 (+ 4602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.900441 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.903197 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.903756 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.903863 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.908035 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:55.908041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:55.908046 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:55.908079 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:55.911549 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.912336 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.912344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.912350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.912515 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.912900 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.912905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.912908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.913279 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:55.913534 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:55.914113 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 02:12:55.914357 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31708,"thread_start_us":33264,"threads_started":1}
I20260929 02:12:55.915277 377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.859588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.24.216.189:34095)
0929 02:12:55.894927 (+ 35339us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:55.894932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:55.894949 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:12:55.894988 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:55.894992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:55.895062 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:55.897353 (+ 2291us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:55.897652 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.899942 (+ 2290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.902751 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.903372 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.908525 (+ 5153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.908731 (+ 206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:55.908737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:55.908742 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:55.908768 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:55.912023 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.912839 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.912844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.912849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.912998 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.913363 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.913367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.913370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.913613 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:55.913831 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:55.914778 (+ 947us) server_negotiation.cc:299] Negotiation successful
0929 02:12:55.914968 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35224,"thread_start_us":39751,"threads_started":1}
I20260929 02:12:55.922257 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.861122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.24.216.188:43899)
0929 02:12:55.898584 (+ 37462us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:55.898591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:55.900635 (+ 2044us) server_negotiation.cc:407] Connection header received
0929 02:12:55.900689 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:55.900695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:55.900762 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:55.900857 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:55.909411 (+ 8554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.910446 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.911849 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.912247 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.912392 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.915585 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:55.915591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:55.915595 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:55.915630 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:55.919474 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.920119 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.920124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.920128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.920264 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:55.920629 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:55.920652 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:55.920655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:55.921005 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:55.921237 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:55.921501 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 02:12:55.921843 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37322,"thread_start_us":37247,"threads_started":1}
I20260929 02:12:55.918797 31579 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:12:55.927201 31579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:12:55.979830 393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.945037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54134 (local address 127.24.216.130:36219)
0929 02:12:55.971542 (+ 26505us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:55.971555 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:55.971595 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:12:55.971713 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:55.971726 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:55.972118 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:55.972949 (+ 831us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:55.974006 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.975226 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.977749 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.978823 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.978996 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.979185 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:55.979407 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 02:12:55.979563 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26303,"thread_start_us":26185,"threads_started":1}
I20260929 02:12:55.980070 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.944248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:36219 (local address 127.0.0.1:54134)
0929 02:12:55.956714 (+ 12466us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:55.956738 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:55.956897 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:55.972911 (+ 16014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:55.972917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:55.972943 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:55.973350 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.973361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.975377 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.975384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.977589 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.977600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.979006 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.979012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.979190 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.979231 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 02:12:55.979344 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 02:12:55.979489 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12326,"mutex_wait_us":31,"thread_start_us":12188,"threads_started":1}
I20260929 02:12:55.982127 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.943803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:39063 (local address 127.0.0.1:50868)
0929 02:12:55.944795 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:55.944827 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:55.944942 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:55.972930 (+ 27988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:55.972933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:55.972949 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:55.973784 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.973805 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.976136 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.976144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.978697 (+ 2553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.978710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.981315 (+ 2605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.981320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.981477 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.981509 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:12:55.981588 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 02:12:55.981712 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":316,"thread_start_us":715,"threads_started":1}
I20260929 02:12:55.982580 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.945524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.24.216.129:38757)
0929 02:12:55.970624 (+ 25100us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:55.970639 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:55.974121 (+ 3482us) server_negotiation.cc:407] Connection header received
0929 02:12:55.974292 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:55.974308 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:55.974556 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:55.974739 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:55.975489 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.977165 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.980068 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.981475 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.981636 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.981784 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:55.982062 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 02:12:55.982257 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24918,"thread_start_us":24816,"threads_started":1}
I20260929 02:12:55.982805 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.944592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:38757 (local address 127.0.0.1:55236)
0929 02:12:55.973996 (+ 29404us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:55.974013 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:55.974127 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:55.974955 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:55.974960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:55.974984 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:55.975257 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.975267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.977308 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.977314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.979504 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:55.979517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.981682 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.981689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:55.981880 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.981908 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:12:55.982002 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 02:12:55.982438 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29271,"mutex_wait_us":19,"thread_start_us":29115,"threads_started":1}
I20260929 02:12:55.983614 392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:55.944744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.24.216.131:39063)
0929 02:12:55.972059 (+ 27315us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:55.972065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:55.972089 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:12:55.972161 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:55.972166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:55.972404 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:55.973230 (+ 826us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:55.974415 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.976003 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.978887 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:55.980919 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:55.981073 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:55.981220 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:55.983181 (+ 1961us) server_negotiation.cc:299] Negotiation successful
0929 02:12:55.983332 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27148,"thread_start_us":26968,"threads_started":1}
I20260929 02:12:55.983841 32624 tablet_service.cc:1467] Processing CreateTablet for tablet 5e20674e0af94bfc954da62793cca5d3 (DEFAULT_TABLE table=test-table [id=914ad9d1042c48b3b62d8720d414e860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:55.985178 32624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e20674e0af94bfc954da62793cca5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:55.986734 32765 tablet_service.cc:1467] Processing CreateTablet for tablet 5e20674e0af94bfc954da62793cca5d3 (DEFAULT_TABLE table=test-table [id=914ad9d1042c48b3b62d8720d414e860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:55.987980 32765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e20674e0af94bfc954da62793cca5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:55.988632 32479 tablet_service.cc:1467] Processing CreateTablet for tablet 5e20674e0af94bfc954da62793cca5d3 (DEFAULT_TABLE table=test-table [id=914ad9d1042c48b3b62d8720d414e860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:55.989869 32479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e20674e0af94bfc954da62793cca5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:55.996815 396 tablet_bootstrap.cc:492] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573: Bootstrap starting.
I20260929 02:12:55.999805 396 tablet_bootstrap.cc:654] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:56.001233 396 log.cc:824] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:56.003185 398 tablet_bootstrap.cc:492] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9: Bootstrap starting.
I20260929 02:12:56.004395 396 tablet_bootstrap.cc:492] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573: No bootstrap required, opened a new log
I20260929 02:12:56.004693 396 ts_tablet_manager.cc:1397] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260929 02:12:56.004854 399 tablet_bootstrap.cc:492] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336: Bootstrap starting.
I20260929 02:12:56.006858 398 tablet_bootstrap.cc:654] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:56.008054 398 log.cc:824] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:56.008333 399 tablet_bootstrap.cc:654] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:56.009526 399 log.cc:824] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336: Log is configured to *not* fsync() on all Append() calls
I20260929 02:12:56.010999 398 tablet_bootstrap.cc:492] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9: No bootstrap required, opened a new log
I20260929 02:12:56.011237 398 ts_tablet_manager.cc:1397] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 02:12:56.011758 399 tablet_bootstrap.cc:492] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336: No bootstrap required, opened a new log
I20260929 02:12:56.011971 399 ts_tablet_manager.cc:1397] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 02:12:56.015386 396 raft_consensus.cc:348] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:56.015836 396 raft_consensus.cc:374] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:56.015937 396 raft_consensus.cc:729] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45cffc9111ec424881246da5734a8573, State: Initialized, Role: FOLLOWER
I20260929 02:12:56.016525 396 consensus_queue.cc:260] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:56.018081 396 ts_tablet_manager.cc:1428] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260929 02:12:56.018739 32690 heartbeater.cc:503] Master 127.24.216.190:39637 was elected leader, sending a full tablet report...
I20260929 02:12:56.020121 399 raft_consensus.cc:348] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:56.020644 399 raft_consensus.cc:374] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:56.020730 399 raft_consensus.cc:729] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1762517f34e643f5a4d2ad8adf685336, State: Initialized, Role: FOLLOWER
I20260929 02:12:56.021340 399 consensus_queue.cc:260] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:56.022977 399 ts_tablet_manager.cc:1428] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 02:12:56.023102 398 raft_consensus.cc:348] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:56.023463 398 raft_consensus.cc:374] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:56.023553 398 raft_consensus.cc:729] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979224aefeeb448287acefed22a220c9, State: Initialized, Role: FOLLOWER
I20260929 02:12:56.024187 398 consensus_queue.cc:260] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:56.027683 363 heartbeater.cc:503] Master 127.24.216.190:39637 was elected leader, sending a full tablet report...
I20260929 02:12:56.028215 398 ts_tablet_manager.cc:1428] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9: Time spent starting tablet: real 0.017s user 0.012s sys 0.002s
W20260929 02:12:56.100958 365 tablet.cc:2369] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:12:56.180881 32549 tablet.cc:2369] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:12:56.241791 32692 tablet.cc:2369] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:12:56.275403 404 raft_consensus.cc:482] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:12:56.275662 404 raft_consensus.cc:504] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:56.277504 404 leader_election.cc:288] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45cffc9111ec424881246da5734a8573 (127.24.216.130:36219), 1762517f34e643f5a4d2ad8adf685336 (127.24.216.129:38757)
I20260929 02:12:56.285543 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:56.277888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:40405 (local address 127.24.216.129:38757)
0929 02:12:56.278142 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:56.278172 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:56.278196 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:12:56.278351 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:56.278356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:56.278429 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:56.278527 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:56.279077 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.280169 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.283237 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.284426 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.284605 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:56.284677 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:56.285156 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 02:12:56.285281 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260929 02:12:56.285543 393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:56.277788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:49133 (local address 127.24.216.130:36219)
0929 02:12:56.277987 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:56.277996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:56.278120 (+ 124us) server_negotiation.cc:407] Connection header received
0929 02:12:56.278352 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:56.278357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:56.278429 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:56.278527 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:56.279081 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.280165 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.283229 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.284468 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.284609 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:56.284677 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:56.285156 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 02:12:56.285281 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:12:56.285686 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:56.277813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:36219 (local address 127.24.216.131:49133)
0929 02:12:56.278028 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:56.278048 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:56.278198 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:56.278559 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:56.278564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:56.278599 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:56.278936 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:56.278947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.280304 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.280308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:56.283078 (+ 2770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:56.283092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.284689 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.284694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:56.284959 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:56.285019 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 02:12:56.285114 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 02:12:56.285298 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":351,"spinlock_wait_cycles":7168}
I20260929 02:12:56.285727 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:56.277771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:38757 (local address 127.24.216.131:40405)
0929 02:12:56.278028 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:56.278048 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:56.278198 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:56.278559 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:56.278564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:56.278599 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:56.278936 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:56.278947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.280303 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.280308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:56.283078 (+ 2770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:56.283092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.284633 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.284656 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:56.284956 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:56.285026 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 02:12:56.285114 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 02:12:56.285298 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":284,"spinlock_wait_cycles":58496}
I20260929 02:12:56.286644 32499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e20674e0af94bfc954da62793cca5d3" candidate_uuid: "979224aefeeb448287acefed22a220c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1762517f34e643f5a4d2ad8adf685336" is_pre_election: true
I20260929 02:12:56.286877 32644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e20674e0af94bfc954da62793cca5d3" candidate_uuid: "979224aefeeb448287acefed22a220c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45cffc9111ec424881246da5734a8573" is_pre_election: true
I20260929 02:12:56.287146 32499 raft_consensus.cc:2454] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 979224aefeeb448287acefed22a220c9 in term 0.
I20260929 02:12:56.287352 32644 raft_consensus.cc:2454] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 979224aefeeb448287acefed22a220c9 in term 0.
I20260929 02:12:56.287829 32720 leader_election.cc:302] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1762517f34e643f5a4d2ad8adf685336, 979224aefeeb448287acefed22a220c9; no voters:
I20260929 02:12:56.288242 404 raft_consensus.cc:2781] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:12:56.288380 404 raft_consensus.cc:482] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:12:56.288491 404 raft_consensus.cc:3037] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:56.290789 404 raft_consensus.cc:504] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:56.291392 404 leader_election.cc:288] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [CANDIDATE]: Term 1 election: Requested vote from peers 45cffc9111ec424881246da5734a8573 (127.24.216.130:36219), 1762517f34e643f5a4d2ad8adf685336 (127.24.216.129:38757)
I20260929 02:12:56.291805 32644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e20674e0af94bfc954da62793cca5d3" candidate_uuid: "979224aefeeb448287acefed22a220c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45cffc9111ec424881246da5734a8573"
I20260929 02:12:56.291824 32499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e20674e0af94bfc954da62793cca5d3" candidate_uuid: "979224aefeeb448287acefed22a220c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1762517f34e643f5a4d2ad8adf685336"
I20260929 02:12:56.291999 32499 raft_consensus.cc:3037] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:56.292001 32644 raft_consensus.cc:3037] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:12:56.295676 32644 raft_consensus.cc:2454] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 979224aefeeb448287acefed22a220c9 in term 1.
I20260929 02:12:56.295674 32499 raft_consensus.cc:2454] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 979224aefeeb448287acefed22a220c9 in term 1.
I20260929 02:12:56.296034 32720 leader_election.cc:302] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1762517f34e643f5a4d2ad8adf685336, 979224aefeeb448287acefed22a220c9; no voters:
I20260929 02:12:56.296350 404 raft_consensus.cc:2781] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:12:56.296685 404 raft_consensus.cc:686] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [term 1 LEADER]: Becoming Leader. State: Replica: 979224aefeeb448287acefed22a220c9, State: Running, Role: LEADER
I20260929 02:12:56.297223 404 consensus_queue.cc:237] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:56.302048 31579 catalog_manager.cc:5495] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 979224aefeeb448287acefed22a220c9 (127.24.216.131). New cstate: current_term: 1 leader_uuid: "979224aefeeb448287acefed22a220c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } health_report { overall_health: UNKNOWN } } }
I20260929 02:12:56.406642 392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:56.401821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.24.216.131:39063)
0929 02:12:56.401982 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:56.401986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:56.402003 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:12:56.402126 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:56.402130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:56.402259 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:56.402381 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:12:56.402675 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.403322 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.404644 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.405031 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.405146 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:56.405549 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:12:56.406078 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:12:56.406252 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:56.406315 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 02:12:56.406372 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:12:56.414753 32644 raft_consensus.cc:1261] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573 [term 1 FOLLOWER]: Refusing update from remote peer 979224aefeeb448287acefed22a220c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:56.414778 32499 raft_consensus.cc:1261] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336 [term 1 FOLLOWER]: Refusing update from remote peer 979224aefeeb448287acefed22a220c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:12:56.415726 405 consensus_queue.cc:1035] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:56.416061 404 consensus_queue.cc:1035] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:12:56.434648 411 mvcc.cc:205] Tried to move back new op lower bound from 7334494111385108480 to 7334494110916722688. Current Snapshot: MvccSnapshot[applied={T|T < 7334494111385108480}]
I20260929 02:12:56.448644 31578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:56.448896 31578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:12:56.454892 31578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75301924e06d4b8682d589278c402ce8: Sending DeleteTablet for 3 replicas of tablet 5e20674e0af94bfc954da62793cca5d3
I20260929 02:12:56.455847 32765 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e20674e0af94bfc954da62793cca5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:56 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:50868
I20260929 02:12:56.456166 32479 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e20674e0af94bfc954da62793cca5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:56 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:55236
I20260929 02:12:56.456544 32624 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e20674e0af94bfc954da62793cca5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:12:56 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:54134
I20260929 02:12:56.466555 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 31547
I20260929 02:12:56.467568 420 tablet_replica.cc:330] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336: stopping tablet replica
I20260929 02:12:56.468106 420 raft_consensus.cc:2229] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:56.468737 420 raft_consensus.cc:2258] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:56.468883 31609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:12:56.469998 421 tablet_replica.cc:330] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573: stopping tablet replica
I20260929 02:12:56.471678 419 tablet_replica.cc:330] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9: stopping tablet replica
I20260929 02:12:56.472213 421 raft_consensus.cc:2229] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:12:56.473418 420 ts_tablet_manager.cc:1905] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:56.473780 419 raft_consensus.cc:2229] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:12:56.474692 419 raft_consensus.cc:2258] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:56.475040 421 raft_consensus.cc:2258] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:12:56.480949 421 ts_tablet_manager.cc:1905] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:56.482803 419 ts_tablet_manager.cc:1905] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:12:56.483778 420 ts_tablet_manager.cc:1918] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:12:56.483986 420 log.cc:1196] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/5e20674e0af94bfc954da62793cca5d3
I20260929 02:12:56.484738 420 ts_tablet_manager.cc:1939] T 5e20674e0af94bfc954da62793cca5d3 P 1762517f34e643f5a4d2ad8adf685336: Deleting consensus metadata
I20260929 02:12:56.496621 419 ts_tablet_manager.cc:1918] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:12:56.496826 419 log.cc:1196] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/5e20674e0af94bfc954da62793cca5d3
I20260929 02:12:56.497463 419 ts_tablet_manager.cc:1939] T 5e20674e0af94bfc954da62793cca5d3 P 979224aefeeb448287acefed22a220c9: Deleting consensus metadata
I20260929 02:12:56.499349 423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:56.458867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.24.216.189:34095)
0929 02:12:56.466559 (+ 7692us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:56.466568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:56.466591 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:12:56.466651 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:56.466706 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:56.466834 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:56.466961 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:56.468954 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.470543 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.473519 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.474334 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.474465 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:56.490205 (+ 15740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:56.490214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:56.490219 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:56.490260 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:56.494622 (+ 4362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:56.495535 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:56.495540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:56.495544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:56.495696 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:56.496122 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:56.496126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:56.496129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:56.496510 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:56.496739 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:56.498547 (+ 1808us) server_negotiation.cc:299] Negotiation successful
0929 02:12:56.498809 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7049,"thread_start_us":8014,"threads_started":1}
I20260929 02:12:56.500098 421 ts_tablet_manager.cc:1918] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:12:56.500259 421 log.cc:1196] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/5e20674e0af94bfc954da62793cca5d3
I20260929 02:12:56.500782 421 ts_tablet_manager.cc:1939] T 5e20674e0af94bfc954da62793cca5d3 P 45cffc9111ec424881246da5734a8573: Deleting consensus metadata
I20260929 02:12:56.517997 424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:56.465971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.24.216.188:43899)
0929 02:12:56.487983 (+ 22012us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:56.487990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:56.502331 (+ 14341us) server_negotiation.cc:407] Connection header received
0929 02:12:56.502400 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:56.502405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:56.502491 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:56.502608 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:56.504284 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.505425 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.507336 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.508183 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.508318 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:56.511675 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:56.511682 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:56.511688 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:56.511722 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:56.515183 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:56.515924 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:56.515930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:56.515934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:56.516073 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:56.516516 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:56.516521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:56.516525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:56.516879 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:56.517124 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:56.517422 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 02:12:56.517628 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21894,"thread_start_us":22016,"threads_started":1}
I20260929 02:12:56.614777 31563 catalog_manager.cc:4841] TS 45cffc9111ec424881246da5734a8573 (127.24.216.130:36219): tablet 5e20674e0af94bfc954da62793cca5d3 (table test-table [id=914ad9d1042c48b3b62d8720d414e860]) successfully deleted
I20260929 02:12:56.614861 31565 catalog_manager.cc:4841] TS 1762517f34e643f5a4d2ad8adf685336 (127.24.216.129:38757): tablet 5e20674e0af94bfc954da62793cca5d3 (table test-table [id=914ad9d1042c48b3b62d8720d414e860]) successfully deleted
I20260929 02:12:56.614814 31566 catalog_manager.cc:4841] TS 979224aefeeb448287acefed22a220c9 (127.24.216.131:39063): tablet 5e20674e0af94bfc954da62793cca5d3 (table test-table [id=914ad9d1042c48b3b62d8720d414e860]) successfully deleted
I20260929 02:12:56.615562 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 31547
I20260929 02:12:56.668706 423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:56.651240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.24.216.189:34095)
0929 02:12:56.651461 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:56.651468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:56.653778 (+ 2310us) server_negotiation.cc:407] Connection header received
0929 02:12:56.654033 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:56.654038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:56.654112 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:56.654198 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:56.656497 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.657582 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.659004 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.659484 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.659623 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:56.662824 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:56.662828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:56.662831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:56.662858 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:56.666018 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:56.666823 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:56.666829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:56.666833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:56.666980 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:56.667395 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:56.667400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:56.667403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:56.667648 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:56.667909 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:56.668207 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 02:12:56.668406 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 02:12:56.670327 424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:56.654095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.24.216.188:43899)
0929 02:12:56.654410 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:56.654416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:56.655015 (+ 599us) server_negotiation.cc:407] Connection header received
0929 02:12:56.655274 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:56.655280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:56.655365 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:56.655470 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:12:56.657219 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.658022 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.660121 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:56.660722 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:56.660859 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:56.664004 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:12:56.664011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:12:56.664017 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:12:56.664056 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:12:56.667209 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:56.667917 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:56.667922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:56.667926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:56.668054 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:12:56.668453 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:12:56.668459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:12:56.668463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:12:56.668774 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:12:56.668986 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:56.669559 (+ 573us) server_negotiation.cc:299] Negotiation successful
0929 02:12:56.670037 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260929 02:12:57.439863 432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.438998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.131:50683)
0929 02:12:57.439518 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:57.439696 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":225,"threads_started":1}
W20260929 02:12:57.440225 363 heartbeater.cc:644] Failed to heartbeat to 127.24.216.190:39637 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
I20260929 02:12:57.444814 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.443483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.130:36373)
0929 02:12:57.444501 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:57.444629 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":786,"threads_started":1}
I20260929 02:12:57.444844 433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.443483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.129:44479)
0929 02:12:57.444563 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:57.444671 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":975,"thread_start_us":894,"threads_started":1}
W20260929 02:12:57.445210 32545 heartbeater.cc:644] Failed to heartbeat to 127.24.216.190:39637 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
W20260929 02:12:57.445238 32690 heartbeater.cc:644] Failed to heartbeat to 127.24.216.190:39637 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
I20260929 02:12:57.954718 435 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 75301924e06d4b8682d589278c402ce8)
I20260929 02:12:57.955037 435 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } }
I20260929 02:12:57.955433 436 raft_consensus.cc:482] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 75301924e06d4b8682d589278c402ce8)
I20260929 02:12:57.955683 436 raft_consensus.cc:504] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } }
I20260929 02:12:57.956568 435 leader_election.cc:288] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75301924e06d4b8682d589278c402ce8 (127.24.216.190:39637), c752a8f94421423dbdd6388c0a1d8845 (127.24.216.189:34095)
I20260929 02:12:57.956921 436 leader_election.cc:288] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75301924e06d4b8682d589278c402ce8 (127.24.216.190:39637), c0e0d77876464385970af94fe0e8c39b (127.24.216.188:43899)
I20260929 02:12:57.957644 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.956637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52614)
0929 02:12:57.957132 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:57.957296 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":440,"threads_started":1}
W20260929 02:12:57.957924 32352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
I20260929 02:12:57.960628 439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.956984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52628)
0929 02:12:57.960053 (+ 3069us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:57.960279 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2924,"thread_start_us":2822,"threads_started":1}
W20260929 02:12:57.960876 32164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
I20260929 02:12:57.972939 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.972250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52632)
0929 02:12:57.972489 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:57.972676 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260929 02:12:57.973182 32352 leader_election.cc:334] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 75301924e06d4b8682d589278c402ce8 (127.24.216.190:39637): Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
I20260929 02:12:57.974905 440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.973929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52634)
0929 02:12:57.974426 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:57.974617 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395}
W20260929 02:12:57.975085 32164 leader_election.cc:334] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 75301924e06d4b8682d589278c402ce8 (127.24.216.190:39637): Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
I20260929 02:12:57.978526 439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.957218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:43899 (local address 127.0.0.1:44672)
0929 02:12:57.960849 (+ 3631us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:57.960873 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:57.961003 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:57.972191 (+ 11188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:57.972196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:57.972225 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:57.972485 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:57.972498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:57.974466 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:57.974472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:57.976604 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:57.976618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:57.977675 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:57.977681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:57.977852 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:57.977897 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 02:12:57.978016 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 02:12:57.978145 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3527,"mutex_wait_us":3,"thread_start_us":16965,"threads_started":1}
I20260929 02:12:57.978601 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.956802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:34095 (local address 127.0.0.1:37538)
0929 02:12:57.957862 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:57.957880 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:57.958046 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:57.972286 (+ 14240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:57.972291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:57.972318 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:57.972668 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:57.972683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:57.974823 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:57.974828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:57.976719 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:57.976734 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:57.977860 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:57.977866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:57.978020 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:57.978048 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:12:57.978177 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 02:12:57.978315 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":11682,"threads_started":1}
I20260929 02:12:57.978703 443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.958297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.24.216.189:34095)
0929 02:12:57.971834 (+ 13537us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:57.971842 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:57.971866 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:12:57.971943 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:57.971950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:57.972038 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:57.972865 (+ 827us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:57.973284 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:57.974642 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:57.976871 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:57.977677 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:57.977860 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:57.977966 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:57.978207 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 02:12:57.978330 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":13290,"thread_start_us":12937,"threads_started":1}
I20260929 02:12:57.978549 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.957480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.24.216.188:43899)
0929 02:12:57.970681 (+ 13201us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:57.970690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:57.970716 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:12:57.970810 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:57.970819 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:57.970921 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:57.972638 (+ 1717us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:57.972900 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:57.974328 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:57.976751 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:57.977505 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:57.977675 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:57.977743 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:57.978041 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 02:12:57.978186 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13036,"thread_start_us":13894,"threads_started":1}
I20260929 02:12:57.979285 32187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0e0d77876464385970af94fe0e8c39b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c752a8f94421423dbdd6388c0a1d8845" is_pre_election: true
I20260929 02:12:57.979521 32375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c752a8f94421423dbdd6388c0a1d8845" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c0e0d77876464385970af94fe0e8c39b" is_pre_election: true
I20260929 02:12:57.979650 32187 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0e0d77876464385970af94fe0e8c39b in term 1.
I20260929 02:12:57.979838 32375 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c752a8f94421423dbdd6388c0a1d8845 in term 1.
I20260929 02:12:57.980229 32350 leader_election.cc:302] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0e0d77876464385970af94fe0e8c39b, c752a8f94421423dbdd6388c0a1d8845; no voters: 75301924e06d4b8682d589278c402ce8
I20260929 02:12:57.980296 32163 leader_election.cc:302] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0e0d77876464385970af94fe0e8c39b, c752a8f94421423dbdd6388c0a1d8845; no voters: 75301924e06d4b8682d589278c402ce8
I20260929 02:12:57.980638 435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 02:12:57.980638 436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 02:12:57.980758 435 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 75301924e06d4b8682d589278c402ce8)
I20260929 02:12:57.980759 436 raft_consensus.cc:482] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 75301924e06d4b8682d589278c402ce8)
I20260929 02:12:57.980858 435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:12:57.980863 436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:12:57.982991 435 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } }
I20260929 02:12:57.983084 436 raft_consensus.cc:504] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } }
I20260929 02:12:57.983537 435 leader_election.cc:288] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [CANDIDATE]: Term 2 election: Requested vote from peers 75301924e06d4b8682d589278c402ce8 (127.24.216.190:39637), c752a8f94421423dbdd6388c0a1d8845 (127.24.216.189:34095)
I20260929 02:12:57.983652 436 leader_election.cc:288] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [CANDIDATE]: Term 2 election: Requested vote from peers 75301924e06d4b8682d589278c402ce8 (127.24.216.190:39637), c0e0d77876464385970af94fe0e8c39b (127.24.216.188:43899)
I20260929 02:12:57.984016 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.983641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52640)
0929 02:12:57.983770 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:57.983870 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 02:12:57.984015 32187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c0e0d77876464385970af94fe0e8c39b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c752a8f94421423dbdd6388c0a1d8845"
I20260929 02:12:57.984287 32187 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c0e0d77876464385970af94fe0e8c39b in current term 2: Already voted for candidate c752a8f94421423dbdd6388c0a1d8845 in this term.
I20260929 02:12:57.984479 439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.983948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52650)
0929 02:12:57.984219 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:57.984304 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 02:12:57.984481 32375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c752a8f94421423dbdd6388c0a1d8845" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c0e0d77876464385970af94fe0e8c39b"
I20260929 02:12:57.984776 32375 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c752a8f94421423dbdd6388c0a1d8845 in current term 2: Already voted for candidate c0e0d77876464385970af94fe0e8c39b in this term.
I20260929 02:12:57.985225 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.984884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52662)
0929 02:12:57.985033 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:57.985115 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 02:12:57.985390 32352 leader_election.cc:334] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 75301924e06d4b8682d589278c402ce8 (127.24.216.190:39637): Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
I20260929 02:12:57.985509 32352 leader_election.cc:302] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0e0d77876464385970af94fe0e8c39b; no voters: 75301924e06d4b8682d589278c402ce8, c752a8f94421423dbdd6388c0a1d8845
I20260929 02:12:57.985677 439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:57.985319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52668)
0929 02:12:57.985473 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:57.985554 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 02:12:57.985795 435 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260929 02:12:57.985872 32164 leader_election.cc:334] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 75301924e06d4b8682d589278c402ce8 (127.24.216.190:39637): Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
I20260929 02:12:57.985984 32164 leader_election.cc:302] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c752a8f94421423dbdd6388c0a1d8845; no voters: 75301924e06d4b8682d589278c402ce8, c0e0d77876464385970af94fe0e8c39b
I20260929 02:12:57.986240 436 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 02:12:58.441511 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:58.440877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.131:46393)
0929 02:12:58.441264 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:58.441340 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":205,"threads_started":1}
I20260929 02:12:58.446555 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:58.445870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.129:58571)
0929 02:12:58.446328 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:58.446398 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":251,"threads_started":1}
I20260929 02:12:58.446628 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:58.445854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.130:34889)
0929 02:12:58.446357 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:58.446444 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"spinlock_wait_cycles":44928,"thread_start_us":272,"threads_started":1}
I20260929 02:12:59.443078 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.442377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.131:59163)
0929 02:12:59.442755 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.442848 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":187,"threads_started":1}
I20260929 02:12:59.448009 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.447388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.130:36431)
0929 02:12:59.447786 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.447862 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":223,"threads_started":1}
I20260929 02:12:59.448390 450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.447388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.129:47069)
0929 02:12:59.448099 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.448221 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":517,"threads_started":1}
I20260929 02:12:59.715898 451 raft_consensus.cc:482] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:12:59.716094 451 raft_consensus.cc:504] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } }
I20260929 02:12:59.716665 451 leader_election.cc:288] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 75301924e06d4b8682d589278c402ce8 (127.24.216.190:39637), c0e0d77876464385970af94fe0e8c39b (127.24.216.188:43899)
I20260929 02:12:59.717180 32375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c752a8f94421423dbdd6388c0a1d8845" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c0e0d77876464385970af94fe0e8c39b" is_pre_election: true
I20260929 02:12:59.717406 32375 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c752a8f94421423dbdd6388c0a1d8845 in term 2.
I20260929 02:12:59.717693 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.716860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52926)
0929 02:12:59.717358 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.717472 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":246,"threads_started":1}
I20260929 02:12:59.717883 32163 leader_election.cc:302] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0e0d77876464385970af94fe0e8c39b, c752a8f94421423dbdd6388c0a1d8845; no voters:
I20260929 02:12:59.718202 451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 02:12:59.718348 451 raft_consensus.cc:482] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:12:59.718458 451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 2 FOLLOWER]: Advancing to term 3
I20260929 02:12:59.719151 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.718810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52940)
0929 02:12:59.718959 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.719043 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:12:59.719415 32164 leader_election.cc:334] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 75301924e06d4b8682d589278c402ce8 (127.24.216.190:39637): Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
I20260929 02:12:59.720997 451 raft_consensus.cc:504] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } }
I20260929 02:12:59.721521 451 leader_election.cc:288] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [CANDIDATE]: Term 3 election: Requested vote from peers 75301924e06d4b8682d589278c402ce8 (127.24.216.190:39637), c0e0d77876464385970af94fe0e8c39b (127.24.216.188:43899)
I20260929 02:12:59.722007 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.721590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52944)
0929 02:12:59.721759 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.721852 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 02:12:59.721980 32375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c752a8f94421423dbdd6388c0a1d8845" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c0e0d77876464385970af94fe0e8c39b"
I20260929 02:12:59.722250 32375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 2 FOLLOWER]: Advancing to term 3
I20260929 02:12:59.722978 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.722621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52958)
0929 02:12:59.722766 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.722851 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:12:59.723150 32164 leader_election.cc:334] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 75301924e06d4b8682d589278c402ce8 (127.24.216.190:39637): Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
I20260929 02:12:59.724551 32375 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c752a8f94421423dbdd6388c0a1d8845 in term 3.
I20260929 02:12:59.724920 32163 leader_election.cc:302] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0e0d77876464385970af94fe0e8c39b, c752a8f94421423dbdd6388c0a1d8845; no voters: 75301924e06d4b8682d589278c402ce8
I20260929 02:12:59.725178 451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 02:12:59.725541 451 raft_consensus.cc:686] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [term 3 LEADER]: Becoming Leader. State: Replica: c752a8f94421423dbdd6388c0a1d8845, State: Running, Role: LEADER
I20260929 02:12:59.725924 451 consensus_queue.cc:237] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } }
I20260929 02:12:59.727751 454 sys_catalog.cc:455] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c752a8f94421423dbdd6388c0a1d8845. Latest consensus state: current_term: 3 leader_uuid: "c752a8f94421423dbdd6388c0a1d8845" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } } }
I20260929 02:12:59.728091 454 sys_catalog.cc:458] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:59.728698 456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:12:59.732170 456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:12:59.732966 456 catalog_manager.cc:1259] Loaded cluster ID: c451746408114b63bc62a23d8e73de81
I20260929 02:12:59.733034 456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:12:59.734208 456 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:12:59.735008 456 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: Loaded TSK: 0
I20260929 02:12:59.735780 456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:12:59.782373 32176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:12:59.789484 32176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:12:59.793510 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.793120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52974)
0929 02:12:59.793302 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.793375 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 02:12:59.793527 32375 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [term 3 FOLLOWER]: Refusing update from remote peer c752a8f94421423dbdd6388c0a1d8845: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 02:12:59.794286 454 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 02:12:59.794588 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.794207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52988)
0929 02:12:59.794376 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.794439 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 02:12:59.794960 32164 consensus_peers.cc:487] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 -> Peer 75301924e06d4b8682d589278c402ce8 (127.24.216.190:39637): Couldn't send request to peer 75301924e06d4b8682d589278c402ce8. Status: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 02:12:59.797158 458 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c752a8f94421423dbdd6388c0a1d8845. Latest consensus state: current_term: 3 leader_uuid: "c752a8f94421423dbdd6388c0a1d8845" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } } }
I20260929 02:12:59.797444 458 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:12:59.798554 451 sys_catalog.cc:455] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c752a8f94421423dbdd6388c0a1d8845" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } } }
I20260929 02:12:59.798874 451 sys_catalog.cc:458] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845 [sys.catalog]: This master's current role is: LEADER
I20260929 02:12:59.799124 458 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c752a8f94421423dbdd6388c0a1d8845" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "75301924e06d4b8682d589278c402ce8" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39637 } } peers { permanent_uuid: "c752a8f94421423dbdd6388c0a1d8845" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 34095 } attrs { promote: false } } peers { permanent_uuid: "c0e0d77876464385970af94fe0e8c39b" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 43899 } attrs { promote: false } } }
I20260929 02:12:59.799409 458 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0e0d77876464385970af94fe0e8c39b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:12:59.799618 459 mvcc.cc:205] Tried to move back new op lower bound from 7334494125228351488 to 7334494124960710656. Current Snapshot: MvccSnapshot[applied={T|T < 7334494125228351488}]
I20260929 02:12:59.818284 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.806914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:38757 (local address 127.0.0.1:44700)
0929 02:12:59.811100 (+ 4186us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.811119 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:59.811259 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:59.811708 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:59.811712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:59.811739 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:59.812068 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:59.812078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:59.813732 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:59.813755 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:59.815773 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:59.815789 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:59.817465 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:59.817471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:59.817667 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:59.817690 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:12:59.817791 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 02:12:59.817910 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4038,"thread_start_us":3897,"threads_started":1}
I20260929 02:12:59.818280 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.807006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.24.216.129:38757)
0929 02:12:59.810633 (+ 3627us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:59.810642 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:59.811216 (+ 574us) server_negotiation.cc:407] Connection header received
0929 02:12:59.811466 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:59.811475 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:59.811578 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:59.811692 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:59.812241 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:59.813575 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:59.816089 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:59.817266 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:59.817462 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:59.817540 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:59.817815 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 02:12:59.817936 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3447,"thread_start_us":3324,"threads_started":1}
I20260929 02:12:59.818881 452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.805994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:36219 (local address 127.0.0.1:49626)
0929 02:12:59.806122 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.806140 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:59.806276 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:59.811392 (+ 5116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:59.811397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:59.811422 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:59.811679 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:59.811691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:59.813467 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:59.813474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:59.815571 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:59.815584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:59.817664 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:59.817671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:59.817844 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:59.817878 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:12:59.817938 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 02:12:59.818067 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 02:12:59.819123 32479 tablet_service.cc:1467] Processing CreateTablet for tablet 089197a5316a4a70ab899d91e71ba002 (DEFAULT_TABLE table=test-table [id=67b58b72076e43eda5d42ded6eb5f4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:59.819613 32479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089197a5316a4a70ab899d91e71ba002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:59.819702 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.806288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.24.216.130:36219)
0929 02:12:59.811084 (+ 4796us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:59.811089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:59.811103 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:12:59.811155 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:59.811158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:59.811245 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:59.811379 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:59.811851 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:59.813330 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:59.815736 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:59.817409 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:59.819141 (+ 1732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:59.819203 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:59.819250 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 02:12:59.819404 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4658,"thread_start_us":275,"threads_started":1}
I20260929 02:12:59.824564 469 tablet_bootstrap.cc:492] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336: Bootstrap starting.
I20260929 02:12:59.824867 32624 tablet_service.cc:1467] Processing CreateTablet for tablet 089197a5316a4a70ab899d91e71ba002 (DEFAULT_TABLE table=test-table [id=67b58b72076e43eda5d42ded6eb5f4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:59.825299 32624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089197a5316a4a70ab899d91e71ba002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:59.825510 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.806637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:39063 (local address 127.0.0.1:37034)
0929 02:12:59.807197 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.807218 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:12:59.807331 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:12:59.818216 (+ 10885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:12:59.818220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:12:59.818241 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:12:59.818574 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:59.818583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:59.821177 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:59.821182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:59.823172 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:12:59.823186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:12:59.824809 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:12:59.824815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:12:59.824972 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:59.825004 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:12:59.825069 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 02:12:59.825188 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":525,"threads_started":1}
I20260929 02:12:59.826313 469 tablet_bootstrap.cc:654] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:59.826392 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.807019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37034 (local address 127.24.216.131:39063)
0929 02:12:59.817839 (+ 10820us) server_negotiation.cc:206] Beginning negotiation
0929 02:12:59.817844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:12:59.817862 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:12:59.817930 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:12:59.817936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:12:59.818004 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:12:59.818091 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:12:59.818740 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:59.821034 (+ 2294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:59.823355 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:12:59.824573 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:12:59.825843 (+ 1270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:12:59.825910 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 02:12:59.825956 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 02:12:59.826074 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10692,"thread_start_us":11149,"threads_started":1}
I20260929 02:12:59.827111 32765 tablet_service.cc:1467] Processing CreateTablet for tablet 089197a5316a4a70ab899d91e71ba002 (DEFAULT_TABLE table=test-table [id=67b58b72076e43eda5d42ded6eb5f4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:12:59.827548 32765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089197a5316a4a70ab899d91e71ba002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:12:59.828466 469 tablet_bootstrap.cc:492] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336: No bootstrap required, opened a new log
I20260929 02:12:59.828598 469 ts_tablet_manager.cc:1397] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:12:59.829355 469 raft_consensus.cc:348] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:59.829617 469 raft_consensus.cc:374] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:59.829738 469 raft_consensus.cc:729] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1762517f34e643f5a4d2ad8adf685336, State: Initialized, Role: FOLLOWER
I20260929 02:12:59.830009 469 consensus_queue.cc:260] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:59.830838 471 tablet_bootstrap.cc:492] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573: Bootstrap starting.
I20260929 02:12:59.832168 471 tablet_bootstrap.cc:654] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:59.833871 471 tablet_bootstrap.cc:492] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573: No bootstrap required, opened a new log
I20260929 02:12:59.834016 471 ts_tablet_manager.cc:1397] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:12:59.834678 469 ts_tablet_manager.cc:1428] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260929 02:12:59.834785 471 raft_consensus.cc:348] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:59.835067 471 raft_consensus.cc:374] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:59.835173 471 raft_consensus.cc:729] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45cffc9111ec424881246da5734a8573, State: Initialized, Role: FOLLOWER
I20260929 02:12:59.835407 471 consensus_queue.cc:260] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:59.835886 450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.835234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.129:56953)
0929 02:12:59.835625 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.835714 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278}
I20260929 02:12:59.839689 471 ts_tablet_manager.cc:1428] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260929 02:12:59.839870 32546 heartbeater.cc:503] Master 127.24.216.189:34095 was elected leader, sending a full tablet report...
I20260929 02:12:59.840829 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.840328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.130:57079)
0929 02:12:59.840579 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.840683 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260929 02:12:59.841463 450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.841057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.129:40865)
0929 02:12:59.841276 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.841356 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 02:12:59.842684 32691 heartbeater.cc:503] Master 127.24.216.189:34095 was elected leader, sending a full tablet report...
I20260929 02:12:59.843432 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.842950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.130:33487)
0929 02:12:59.843109 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.843216 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 02:12:59.846138 474 tablet_bootstrap.cc:492] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9: Bootstrap starting.
I20260929 02:12:59.848162 474 tablet_bootstrap.cc:654] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9: Neither blocks nor log segments found. Creating new log.
I20260929 02:12:59.849817 474 tablet_bootstrap.cc:492] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9: No bootstrap required, opened a new log
I20260929 02:12:59.849937 474 ts_tablet_manager.cc:1397] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:12:59.850581 474 raft_consensus.cc:348] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:59.850766 474 raft_consensus.cc:374] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:12:59.850831 474 raft_consensus.cc:729] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979224aefeeb448287acefed22a220c9, State: Initialized, Role: FOLLOWER
I20260929 02:12:59.851047 474 consensus_queue.cc:260] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:12:59.851701 474 ts_tablet_manager.cc:1428] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:12:59.852376 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.851987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.131:47317)
0929 02:12:59.852138 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.852221 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 02:12:59.853416 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:12:59.853021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.131:56911)
0929 02:12:59.853173 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 02:12:59.853238 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 02:12:59.853708 364 heartbeater.cc:503] Master 127.24.216.189:34095 was elected leader, sending a full tablet report...
I20260929 02:13:00.103010 472 raft_consensus.cc:482] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:13:00.103331 472 raft_consensus.cc:504] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:13:00.104848 472 leader_election.cc:288] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45cffc9111ec424881246da5734a8573 (127.24.216.130:36219), 979224aefeeb448287acefed22a220c9 (127.24.216.131:39063)
I20260929 02:13:00.112984 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.105239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:54389 (local address 127.24.216.131:39063)
0929 02:13:00.105452 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:00.105456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:00.105726 (+ 270us) server_negotiation.cc:407] Connection header received
0929 02:13:00.105949 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:00.105958 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:00.106016 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:00.106133 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:13:00.106793 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.107904 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.110751 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.111928 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.112082 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:00.112149 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:00.112608 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 02:13:00.112726 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:13:00.112974 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.105325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:37305 (local address 127.24.216.130:36219)
0929 02:13:00.105566 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:00.105575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:00.106078 (+ 503us) server_negotiation.cc:407] Connection header received
0929 02:13:00.106256 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:00.106261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:00.106331 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:00.106428 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:13:00.107047 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.108080 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.110752 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.111938 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.112080 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:00.112149 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:00.112608 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 02:13:00.112726 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:13:00.113121 450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.105153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:39063 (local address 127.24.216.129:54389)
0929 02:13:00.105590 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:00.105617 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:00.105783 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:00.106190 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:00.106197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:00.106236 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:13:00.106604 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:00.106621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.108041 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.108045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:00.110603 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:00.110618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.112143 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.112147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:00.112401 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:00.112480 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 02:13:00.112569 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 02:13:00.112718 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":284}
I20260929 02:13:00.113160 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.105241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:36219 (local address 127.24.216.129:37305)
0929 02:13:00.105987 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:00.105999 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:00.106085 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:00.106490 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:00.106496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:00.106518 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:13:00.106902 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:00.106914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.108176 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.108182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:00.110603 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:00.110618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.112126 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.112133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:00.112400 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:00.112475 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 02:13:00.112569 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 02:13:00.112718 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":150,"thread_start_us":391,"threads_started":1}
I20260929 02:13:00.113937 32644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089197a5316a4a70ab899d91e71ba002" candidate_uuid: "1762517f34e643f5a4d2ad8adf685336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45cffc9111ec424881246da5734a8573" is_pre_election: true
I20260929 02:13:00.114200 32644 raft_consensus.cc:2454] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1762517f34e643f5a4d2ad8adf685336 in term 0.
I20260929 02:13:00.114091 317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089197a5316a4a70ab899d91e71ba002" candidate_uuid: "1762517f34e643f5a4d2ad8adf685336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "979224aefeeb448287acefed22a220c9" is_pre_election: true
I20260929 02:13:00.114529 317 raft_consensus.cc:2454] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1762517f34e643f5a4d2ad8adf685336 in term 0.
I20260929 02:13:00.114699 32432 leader_election.cc:302] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1762517f34e643f5a4d2ad8adf685336, 45cffc9111ec424881246da5734a8573; no voters:
I20260929 02:13:00.115039 472 raft_consensus.cc:2781] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:13:00.115161 472 raft_consensus.cc:482] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:13:00.115265 472 raft_consensus.cc:3037] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:00.117273 472 raft_consensus.cc:504] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:13:00.117838 472 leader_election.cc:288] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [CANDIDATE]: Term 1 election: Requested vote from peers 45cffc9111ec424881246da5734a8573 (127.24.216.130:36219), 979224aefeeb448287acefed22a220c9 (127.24.216.131:39063)
I20260929 02:13:00.118222 32644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089197a5316a4a70ab899d91e71ba002" candidate_uuid: "1762517f34e643f5a4d2ad8adf685336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45cffc9111ec424881246da5734a8573"
I20260929 02:13:00.118283 317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089197a5316a4a70ab899d91e71ba002" candidate_uuid: "1762517f34e643f5a4d2ad8adf685336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "979224aefeeb448287acefed22a220c9"
I20260929 02:13:00.118403 32644 raft_consensus.cc:3037] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:00.118495 317 raft_consensus.cc:3037] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:00.120549 32644 raft_consensus.cc:2454] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1762517f34e643f5a4d2ad8adf685336 in term 1.
I20260929 02:13:00.120651 317 raft_consensus.cc:2454] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1762517f34e643f5a4d2ad8adf685336 in term 1.
I20260929 02:13:00.120919 32432 leader_election.cc:302] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1762517f34e643f5a4d2ad8adf685336, 45cffc9111ec424881246da5734a8573; no voters:
I20260929 02:13:00.121178 472 raft_consensus.cc:2781] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:13:00.121608 472 raft_consensus.cc:686] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [term 1 LEADER]: Becoming Leader. State: Replica: 1762517f34e643f5a4d2ad8adf685336, State: Running, Role: LEADER
I20260929 02:13:00.122022 472 consensus_queue.cc:237] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } } peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } }
I20260929 02:13:00.122853 450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.122394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.129:49333)
0929 02:13:00.122625 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:00.122694 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 02:13:00.125782 32176 catalog_manager.cc:5495] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1762517f34e643f5a4d2ad8adf685336 (127.24.216.129). New cstate: current_term: 1 leader_uuid: "1762517f34e643f5a4d2ad8adf685336" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1762517f34e643f5a4d2ad8adf685336" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 38757 } health_report { overall_health: HEALTHY } } }
I20260929 02:13:00.180177 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.175709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44714 (local address 127.24.216.129:38757)
0929 02:13:00.175881 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:00.175885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:00.175901 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:13:00.176009 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:00.176013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:00.176092 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:00.176224 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:13:00.176555 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.177261 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.178442 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.178820 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.178907 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:00.179215 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:13:00.179756 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:13:00.179870 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:00.179928 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 02:13:00.179984 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:13:00.184057 32644 raft_consensus.cc:1261] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573 [term 1 FOLLOWER]: Refusing update from remote peer 1762517f34e643f5a4d2ad8adf685336: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:13:00.184314 317 raft_consensus.cc:1261] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9 [term 1 FOLLOWER]: Refusing update from remote peer 1762517f34e643f5a4d2ad8adf685336: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:13:00.184808 479 consensus_queue.cc:1035] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45cffc9111ec424881246da5734a8573" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:13:00.185259 472 consensus_queue.cc:1035] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "979224aefeeb448287acefed22a220c9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:13:00.187434 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.187018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.131:59017)
0929 02:13:00.187183 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:00.187275 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 02:13:00.187798 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.187370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.130:58033)
0929 02:13:00.187524 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:00.187603 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 02:13:00.191620 482 mvcc.cc:205] Tried to move back new op lower bound from 7334494126827474944 to 7334494126584102912. Current Snapshot: MvccSnapshot[applied={T|T < 7334494126827474944}]
I20260929 02:13:00.193511 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.192740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.130:43361)
0929 02:13:00.193172 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:00.193283 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248}
I20260929 02:13:00.196077 450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.195351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.129:50237)
0929 02:13:00.195874 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:00.195948 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431}
I20260929 02:13:00.204404 450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.203900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.129:59355)
0929 02:13:00.204117 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:00.204200 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 02:13:00.206581 448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.206026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.24.216.131:50489)
0929 02:13:00.206284 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:00.206383 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 02:13:00.207617 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.207081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:52996)
0929 02:13:00.207331 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:00.207433 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260929 02:13:00.209431 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.208883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:39637 (local address 127.0.0.1:53010)
0929 02:13:00.209120 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:00.209229 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:39637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 02:13:00.209772 484 mvcc.cc:205] Tried to move back new op lower bound from 7334494126827474944 to 7334494126584102912. Current Snapshot: MvccSnapshot[applied={T|T < 7334494126827474944}]
I20260929 02:13:00.216598 32176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:13:00.216825 32176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:13:00.222245 32176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c752a8f94421423dbdd6388c0a1d8845: Sending DeleteTablet for 3 replicas of tablet 089197a5316a4a70ab899d91e71ba002
I20260929 02:13:00.223500 32765 tablet_service.cc:1514] Processing DeleteTablet for tablet 089197a5316a4a70ab899d91e71ba002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:00 UTC) from {username='oryx', principal='oryx/127.24.216.189@KRBTEST.COM'} at 127.0.0.1:37034
I20260929 02:13:00.223685 32624 tablet_service.cc:1514] Processing DeleteTablet for tablet 089197a5316a4a70ab899d91e71ba002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:00 UTC) from {username='oryx', principal='oryx/127.24.216.189@KRBTEST.COM'} at 127.0.0.1:49626
I20260929 02:13:00.223930 32479 tablet_service.cc:1514] Processing DeleteTablet for tablet 089197a5316a4a70ab899d91e71ba002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:00 UTC) from {username='oryx', principal='oryx/127.24.216.189@KRBTEST.COM'} at 127.0.0.1:44700
I20260929 02:13:00.224151 492 tablet_replica.cc:330] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9: stopping tablet replica
I20260929 02:13:00.224339 492 raft_consensus.cc:2229] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:13:00.224603 492 raft_consensus.cc:2258] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:00.225277 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 32333
I20260929 02:13:00.225600 492 ts_tablet_manager.cc:1905] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:00.233559 492 ts_tablet_manager.cc:1918] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:13:00.233780 492 log.cc:1196] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/089197a5316a4a70ab899d91e71ba002
I20260929 02:13:00.234283 492 ts_tablet_manager.cc:1939] T 089197a5316a4a70ab899d91e71ba002 P 979224aefeeb448287acefed22a220c9: Deleting consensus metadata
I20260929 02:13:00.236090 494 tablet_replica.cc:330] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336: stopping tablet replica
I20260929 02:13:00.236385 494 raft_consensus.cc:2229] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:13:00.236483 493 tablet_replica.cc:330] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573: stopping tablet replica
I20260929 02:13:00.236759 493 raft_consensus.cc:2229] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:13:00.237057 494 raft_consensus.cc:2258] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:00.237155 493 raft_consensus.cc:2258] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:00.238461 494 ts_tablet_manager.cc:1905] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:00.239297 493 ts_tablet_manager.cc:1905] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:00.245103 494 ts_tablet_manager.cc:1918] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:13:00.246991 32164 catalog_manager.cc:4841] TS 979224aefeeb448287acefed22a220c9 (127.24.216.131:39063): tablet 089197a5316a4a70ab899d91e71ba002 (table test-table [id=67b58b72076e43eda5d42ded6eb5f4b8]) successfully deleted
I20260929 02:13:00.248507 494 log.cc:1196] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/089197a5316a4a70ab899d91e71ba002
I20260929 02:13:00.248817 493 ts_tablet_manager.cc:1918] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:13:00.249022 493 log.cc:1196] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/089197a5316a4a70ab899d91e71ba002
I20260929 02:13:00.249043 494 ts_tablet_manager.cc:1939] T 089197a5316a4a70ab899d91e71ba002 P 1762517f34e643f5a4d2ad8adf685336: Deleting consensus metadata
I20260929 02:13:00.249552 493 ts_tablet_manager.cc:1939] T 089197a5316a4a70ab899d91e71ba002 P 45cffc9111ec424881246da5734a8573: Deleting consensus metadata
I20260929 02:13:00.250700 32161 catalog_manager.cc:4841] TS 45cffc9111ec424881246da5734a8573 (127.24.216.130:36219): tablet 089197a5316a4a70ab899d91e71ba002 (table test-table [id=67b58b72076e43eda5d42ded6eb5f4b8]) successfully deleted
I20260929 02:13:00.253818 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.228213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.24.216.129:38757)
0929 02:13:00.228425 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:00.228434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:00.237902 (+ 9468us) server_negotiation.cc:407] Connection header received
0929 02:13:00.237968 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:00.237973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:00.238051 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:00.238179 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:00.239943 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.240687 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.242443 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.242914 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.243067 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:00.246513 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:00.246532 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:00.246556 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:00.246599 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:00.251047 (+ 4448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:00.251725 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:00.251745 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:00.251748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:00.251887 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:00.252273 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:00.252279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:00.252282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:00.252616 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:00.252940 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:00.253214 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 02:13:00.253437 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:13:00.254518 32539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:00.355715 32163 catalog_manager.cc:4841] TS 1762517f34e643f5a4d2ad8adf685336 (127.24.216.129:38757): tablet 089197a5316a4a70ab899d91e71ba002 (table test-table [id=67b58b72076e43eda5d42ded6eb5f4b8]) successfully deleted
I20260929 02:13:00.356081 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 32333
I20260929 02:13:00.374769 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 32556
I20260929 02:13:00.387333 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.375491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49640 (local address 127.24.216.130:36219)
0929 02:13:00.375690 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:00.375694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:00.375773 (+ 79us) server_negotiation.cc:407] Connection header received
0929 02:13:00.375910 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:00.375915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:00.375977 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:00.376067 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:00.377387 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.378076 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.379277 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.379673 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.379773 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:00.382133 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:00.382186 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:00.382213 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:00.382242 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:00.384980 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:00.385588 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:00.385596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:00.385598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:00.385737 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:00.386044 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:00.386047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:00.386049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:00.386420 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:00.386631 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:00.386835 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 02:13:00.386979 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:13:00.388007 32684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:00.479403 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 32556
I20260929 02:13:00.499130 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 32702
I20260929 02:13:00.511448 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:00.499921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.24.216.131:39063)
0929 02:13:00.500115 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:00.500121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:00.500230 (+ 109us) server_negotiation.cc:407] Connection header received
0929 02:13:00.500428 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:00.500433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:00.500494 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:00.500577 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:00.501837 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.502575 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.503706 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:00.504074 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:00.504163 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:00.506516 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:00.506529 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:00.506545 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:00.506582 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:00.509263 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:00.509823 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:00.509831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:00.509833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:00.509959 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:00.510312 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:00.510315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:00.510317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:00.510634 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:00.510870 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:00.511060 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 02:13:00.511205 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:13:00.512063 357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:00.628198 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 32702
I20260929 02:13:00.649996 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 32041
I20260929 02:13:00.651089 32207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:00.769505 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 32041
I20260929 02:13:00.790552 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 32141
I20260929 02:13:00.791572 32395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:00.902089 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 32141
2026-09-29T02:13:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11496 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:13:00 dist-test-slave-9k0h krb5kdc[503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:13:00 dist-test-slave-9k0h krb5kdc[503](info): set up 2 sockets
Sep 29 02:13:00 dist-test-slave-9k0h krb5kdc[503](info): commencing operation
krb5kdc: starting...
W20260929 02:13:01.994997 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:13:02 dist-test-slave-9k0h krb5kdc[503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:13:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:13:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.190.keytab.
Entry for principal oryx/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.190.keytab.
I20260929 02:13:02.208765 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:40503
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.190.keytab
--principal=oryx/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:40503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:13:02.526340 516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:02.526676 516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:02.526782 516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:02.533764 516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:13:02.533896 516 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:13:02.533975 516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:02.534078 516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:02.534176 516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:13:02.534276 516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:13:02.543655 516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:40503
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.190.keytab
--principal=oryx/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:40503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:02.544785 516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:02.546002 516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:02.557433 524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:02.557468 521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:02.557435 522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:02.558543 516 server_base.cc:1034] running on GCE node
I20260929 02:13:02.559231 516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:02.560520 516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:02.561758 516 hybrid_clock.cc:648] HybridClock initialized: now 1790647982561706 us; error 49 us; skew 500 ppm
I20260929 02:13:02.562068 516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:02 dist-test-slave-9k0h krb5kdc[503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647982, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:02.564946 516 init.cc:377] Logged in from keytab as oryx/127.24.216.190@KRBTEST.COM (short username oryx)
I20260929 02:13:02.566411 516 webserver.cc:460] Webserver started at http://127.24.216.190:44197/ using document root <none> and password file <none>
I20260929 02:13:02.567101 516 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:02.567250 516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:02.567559 516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:02.570418 516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "af83698ec90144a08c569fb8763414b7"
format_stamp: "Formatted at 2026-09-29 02:13:02 on dist-test-slave-9k0h"
server_key: "18b3804b5dd3fbc45a10e3eccfbc1571"
server_key_iv: "b4fb602a94d55dd84ef3d3d59e2cb423"
server_key_version: "encryptionkey@0"
I20260929 02:13:02.571169 516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "af83698ec90144a08c569fb8763414b7"
format_stamp: "Formatted at 2026-09-29 02:13:02 on dist-test-slave-9k0h"
server_key: "18b3804b5dd3fbc45a10e3eccfbc1571"
server_key_iv: "b4fb602a94d55dd84ef3d3d59e2cb423"
server_key_version: "encryptionkey@0"
I20260929 02:13:02.576439 516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:13:02.579721 531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:02.580876 516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 02:13:02.581059 516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "af83698ec90144a08c569fb8763414b7"
format_stamp: "Formatted at 2026-09-29 02:13:02 on dist-test-slave-9k0h"
server_key: "18b3804b5dd3fbc45a10e3eccfbc1571"
server_key_iv: "b4fb602a94d55dd84ef3d3d59e2cb423"
server_key_version: "encryptionkey@0"
I20260929 02:13:02.581259 516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:02.610766 516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:02.614542 516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:02.614820 516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:02.635514 516 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:40503
I20260929 02:13:02.635532 592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:40503 every 8 connection(s)
I20260929 02:13:02.636837 516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:13:02.638556 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 516
I20260929 02:13:02.638779 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:13:02.639407 25442 external_mini_cluster.cc:1428] Setting key 3299aa6177f9d1ee703ac9c6e5963f5b
I20260929 02:13:02.640875 593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:13:02 dist-test-slave-9k0h krb5kdc[503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
I20260929 02:13:02.654531 593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7: Bootstrap starting.
I20260929 02:13:02.657469 593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:02.658552 593 log.cc:824] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:13:02.661329 593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7: No bootstrap required, opened a new log
I20260929 02:13:02.661563 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:02.641381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.24.216.190:40503)
0929 02:13:02.642232 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:02.642252 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:02.642298 (+ 46us) server_negotiation.cc:407] Connection header received
0929 02:13:02.643795 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:02.643815 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:02.644182 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:02.644502 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:02.646276 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:02.647587 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:02.649300 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:02.649918 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:02.650037 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:02.654661 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:02.654678 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:02.654692 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:02.654724 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:02.657564 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:02.658309 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:02.658316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:02.658322 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:02.658473 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:02.658924 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:02.658928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:02.658930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:02.659484 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:02.659787 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:02.660215 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 02:13:02.660593 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":373,"threads_started":1}
I20260929 02:13:02.668622 593 raft_consensus.cc:348] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af83698ec90144a08c569fb8763414b7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 40503 } }
I20260929 02:13:02.669056 593 raft_consensus.cc:374] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:02.669171 593 raft_consensus.cc:729] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af83698ec90144a08c569fb8763414b7, State: Initialized, Role: FOLLOWER
I20260929 02:13:02.669765 593 consensus_queue.cc:260] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af83698ec90144a08c569fb8763414b7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 40503 } }
I20260929 02:13:02.670006 593 raft_consensus.cc:388] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:13:02.670140 593 raft_consensus.cc:482] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:13:02.670348 593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:02.672333 593 raft_consensus.cc:504] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af83698ec90144a08c569fb8763414b7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 40503 } }
I20260929 02:13:02.672787 593 leader_election.cc:302] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af83698ec90144a08c569fb8763414b7; no voters:
I20260929 02:13:02.673388 593 leader_election.cc:288] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:13:02.673465 598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:13:02.674191 598 raft_consensus.cc:686] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [term 1 LEADER]: Becoming Leader. State: Replica: af83698ec90144a08c569fb8763414b7, State: Running, Role: LEADER
I20260929 02:13:02.674631 598 consensus_queue.cc:237] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af83698ec90144a08c569fb8763414b7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 40503 } }
I20260929 02:13:02.675304 593 sys_catalog.cc:564] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:13:02.677297 600 sys_catalog.cc:455] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af83698ec90144a08c569fb8763414b7. Latest consensus state: current_term: 1 leader_uuid: "af83698ec90144a08c569fb8763414b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af83698ec90144a08c569fb8763414b7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 40503 } } }
I20260929 02:13:02.677594 600 sys_catalog.cc:458] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:13:02.677178 599 sys_catalog.cc:455] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af83698ec90144a08c569fb8763414b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af83698ec90144a08c569fb8763414b7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 40503 } } }
I20260929 02:13:02.677901 599 sys_catalog.cc:458] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:13:02.679767 603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:13:02.684513 603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:13:02.693822 603 catalog_manager.cc:1347] Generated new cluster ID: 367681a4ed554719884d6446bc13978a
I20260929 02:13:02.693924 603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:13:02.711288 603 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:13:02.712563 603 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:13:02.726573 603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7: Generated new TSK 0
I20260929 02:13:02.727454 603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.129.keytab.
Entry for principal oryx/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.129.keytab.
I20260929 02:13:02.778040 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:40503
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.129.keytab
--principal=oryx/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:13:03.088718 619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:03.088979 619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:03.089133 619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:03.096120 619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:03.096239 619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:03.096379 619 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:13:03.096491 619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:13:03.105509 619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.129.keytab
--principal=oryx/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.216.190:40503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:03.106760 619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:03.108006 619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:03.120234 625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:03.120215 624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:03.120167 627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:03.120911 619 server_base.cc:1034] running on GCE node
I20260929 02:13:03.121472 619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:03.122393 619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:03.123600 619 hybrid_clock.cc:648] HybridClock initialized: now 1790647983123551 us; error 38 us; skew 500 ppm
I20260929 02:13:03.123888 619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:03 dist-test-slave-9k0h krb5kdc[503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647983, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:03.126922 619 init.cc:377] Logged in from keytab as oryx/127.24.216.129@KRBTEST.COM (short username oryx)
I20260929 02:13:03.128168 619 webserver.cc:460] Webserver started at http://127.24.216.129:39967/ using document root <none> and password file <none>
I20260929 02:13:03.128826 619 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:03.128906 619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:03.129158 619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:03.131860 619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "ca6faaba97bf4b9691d4c6536df17c88"
format_stamp: "Formatted at 2026-09-29 02:13:03 on dist-test-slave-9k0h"
server_key: "be9a1480b4052b6b6c9ecff87554562c"
server_key_iv: "b629573156f8493d4c8cc3b2ee525ebc"
server_key_version: "encryptionkey@0"
I20260929 02:13:03.132586 619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "ca6faaba97bf4b9691d4c6536df17c88"
format_stamp: "Formatted at 2026-09-29 02:13:03 on dist-test-slave-9k0h"
server_key: "be9a1480b4052b6b6c9ecff87554562c"
server_key_iv: "b629573156f8493d4c8cc3b2ee525ebc"
server_key_version: "encryptionkey@0"
I20260929 02:13:03.137442 619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:13:03.140686 634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:03.141952 619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:13:03.142072 619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "ca6faaba97bf4b9691d4c6536df17c88"
format_stamp: "Formatted at 2026-09-29 02:13:03 on dist-test-slave-9k0h"
server_key: "be9a1480b4052b6b6c9ecff87554562c"
server_key_iv: "b629573156f8493d4c8cc3b2ee525ebc"
server_key_version: "encryptionkey@0"
I20260929 02:13:03.142360 619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:03.169421 619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:03.173730 619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:03.174031 619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:03.176790 619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:03.176895 619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:03.177069 619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:03.177160 619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:13:03 dist-test-slave-9k0h krb5kdc[503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647983, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
I20260929 02:13:03.200289 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:03.178184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:40503 (local address 127.24.216.129:45163)
0929 02:13:03.179062 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:03.179107 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:03.181035 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:03.181749 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:03.181763 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:03.182582 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:03.183353 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:03.183397 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.184988 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.184994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:03.187283 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:03.187297 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.188116 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.188127 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:03.188465 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:03.189172 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:03.189209 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:03.192534 (+ 3325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:03.196334 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:03.196348 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:03.196366 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:03.196976 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:03.197403 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:03.197408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:03.197411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:03.197632 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:03.198137 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:03.198143 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:03.198585 (+ 442us) client_negotiation.cc:769] Sending connection context
0929 02:13:03.198923 (+ 338us) client_negotiation.cc:240] Negotiation successful
0929 02:13:03.199285 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":378,"threads_started":1}
I20260929 02:13:03.201225 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:03.177546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:45163 (local address 127.24.216.190:40503)
0929 02:13:03.178044 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:03.178049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:03.179231 (+ 1182us) server_negotiation.cc:407] Connection header received
0929 02:13:03.181290 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:03.181296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:03.181383 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:03.181504 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:03.183561 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.184808 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.187432 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.187847 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.187981 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:03.192775 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:03.192796 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:03.192802 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:03.192848 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:03.196155 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:03.197169 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:03.197174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:03.197177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:03.197290 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:03.197744 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:03.197749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:03.197752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:03.198008 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:03.198158 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:03.200758 (+ 2600us) server_negotiation.cc:299] Negotiation successful
0929 02:13:03.200964 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":276,"threads_started":1}
I20260929 02:13:03.221680 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:03.207222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:45297 (local address 127.24.216.190:40503)
0929 02:13:03.207443 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:03.207451 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:03.207987 (+ 536us) server_negotiation.cc:407] Connection header received
0929 02:13:03.208113 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:03.208121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:03.208211 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:03.208332 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:03.209122 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.210516 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.212265 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.212658 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.212778 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:03.214632 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:03.214639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:03.214647 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:03.214683 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:03.218398 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:03.219306 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:03.219332 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:03.219338 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:03.219559 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:03.220060 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:03.220067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:03.220071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:03.220406 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:03.220570 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:03.221147 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 02:13:03.221344 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:13:03.222816 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:03.207187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:40503 (local address 127.24.216.129:45297)
0929 02:13:03.207888 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:03.207906 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:03.208009 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:03.208342 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:03.208345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:03.208727 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:03.208994 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:03.209002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.210693 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.210701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:03.212131 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:03.212143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.212838 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.212845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:03.213012 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:03.213552 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:03.213565 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:03.214419 (+ 854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:03.218558 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:03.218568 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:03.218574 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:03.219095 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:03.219713 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:03.219720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:03.219726 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:03.219912 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:03.220646 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:03.220652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:03.220870 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 02:13:03.221938 (+ 1068us) client_negotiation.cc:240] Negotiation successful
0929 02:13:03.222314 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":425,"threads_started":1}
I20260929 02:13:03.239699 619 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:46435
I20260929 02:13:03.239722 754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:46435 every 8 connection(s)
I20260929 02:13:03.240885 619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:13:03.242764 755 heartbeater.cc:347] Connected to a master server at 127.24.216.190:40503
I20260929 02:13:03.242982 755 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:03.243556 755 heartbeater.cc:511] Master 127.24.216.190:40503 requested a full tablet report, sending...
I20260929 02:13:03.245371 548 ts_manager.cc:194] Registered new tserver with Master: ca6faaba97bf4b9691d4c6536df17c88 (127.24.216.129:46435)
I20260929 02:13:03.246897 548 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:45163
I20260929 02:13:03.248615 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 619
I20260929 02:13:03.248821 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:13:03.249444 25442 external_mini_cluster.cc:1428] Setting key 94b03eaa9e2f014146b4e5d25f7e7c06
WARNING: no policy specified for oryx/127.24.216.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.130.keytab.
Entry for principal oryx/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.130.keytab.
I20260929 02:13:03.296934 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:40503
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.130.keytab
--principal=oryx/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:13:03.603983 759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:03.604291 759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:03.604470 759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:03.611507 759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:03.611639 759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:03.611779 759 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:13:03.611912 759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:13:03.621177 759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.130.keytab
--principal=oryx/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.216.190:40503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:03.622424 759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:03.623706 759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:03.636265 765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:03.636296 767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:03.636503 764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:03.637434 759 server_base.cc:1034] running on GCE node
I20260929 02:13:03.638008 759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:03.638932 759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:03.640170 759 hybrid_clock.cc:648] HybridClock initialized: now 1790647983640102 us; error 51 us; skew 500 ppm
I20260929 02:13:03.640475 759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:03 dist-test-slave-9k0h krb5kdc[503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647983, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:03.643739 759 init.cc:377] Logged in from keytab as oryx/127.24.216.130@KRBTEST.COM (short username oryx)
I20260929 02:13:03.645299 759 webserver.cc:460] Webserver started at http://127.24.216.130:39425/ using document root <none> and password file <none>
I20260929 02:13:03.646056 759 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:03.646210 759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:03.646597 759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:03.649590 759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "4133caddbdd84b8dbc4ea2f3fff2de45"
format_stamp: "Formatted at 2026-09-29 02:13:03 on dist-test-slave-9k0h"
server_key: "3ccfaaebf8afcf242cec571d077857aa"
server_key_iv: "b6fb159eebd41b565b55e858f0e6e750"
server_key_version: "encryptionkey@0"
I20260929 02:13:03.650525 759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "4133caddbdd84b8dbc4ea2f3fff2de45"
format_stamp: "Formatted at 2026-09-29 02:13:03 on dist-test-slave-9k0h"
server_key: "3ccfaaebf8afcf242cec571d077857aa"
server_key_iv: "b6fb159eebd41b565b55e858f0e6e750"
server_key_version: "encryptionkey@0"
I20260929 02:13:03.655805 759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:13:03.659253 774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:03.660570 759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:13:03.660737 759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "4133caddbdd84b8dbc4ea2f3fff2de45"
format_stamp: "Formatted at 2026-09-29 02:13:03 on dist-test-slave-9k0h"
server_key: "3ccfaaebf8afcf242cec571d077857aa"
server_key_iv: "b6fb159eebd41b565b55e858f0e6e750"
server_key_version: "encryptionkey@0"
I20260929 02:13:03.660957 759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:03.683820 759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:03.687976 759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:03.688248 759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:03.690896 759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:03.691026 759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:03.691169 759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:03.691239 759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:13:03 dist-test-slave-9k0h krb5kdc[503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647983, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
I20260929 02:13:03.715569 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:03.691660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:55239 (local address 127.24.216.190:40503)
0929 02:13:03.691871 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:03.691876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:03.693547 (+ 1671us) server_negotiation.cc:407] Connection header received
0929 02:13:03.695859 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:03.695868 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:03.695975 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:03.696111 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:03.698176 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.699512 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.702020 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.702810 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.702976 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:03.708754 (+ 5778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:03.708762 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:03.708769 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:03.708828 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:03.711924 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:03.712813 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:03.712821 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:03.712827 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:03.712949 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:03.713472 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:03.713478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:03.713481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:03.713890 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:03.714038 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:03.715003 (+ 965us) server_negotiation.cc:299] Negotiation successful
0929 02:13:03.715198 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:13:03.717132 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:03.692606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:40503 (local address 127.24.216.130:55239)
0929 02:13:03.693397 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:03.693435 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:03.695457 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:03.696343 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:03.696355 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:03.697116 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:03.697945 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:03.697975 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.699699 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.699705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:03.701850 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:03.701868 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.703340 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.703350 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:03.703705 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:03.704557 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:03.704604 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:03.708491 (+ 3887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:03.712045 (+ 3554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:03.712053 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:03.712074 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:03.712681 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:03.713064 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:03.713071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:03.713079 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:03.713370 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:03.714119 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:03.714124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:03.714641 (+ 517us) client_negotiation.cc:769] Sending connection context
0929 02:13:03.714920 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 02:13:03.715445 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":327,"threads_started":1}
I20260929 02:13:03.733122 759 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:42635
I20260929 02:13:03.733160 893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:42635 every 8 connection(s)
I20260929 02:13:03.734683 759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:13:03.737309 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 759
I20260929 02:13:03.737462 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:13:03.737995 25442 external_mini_cluster.cc:1428] Setting key 16e580c1d285e50e06c67d372d527d80
I20260929 02:13:03.736819 894 heartbeater.cc:347] Connected to a master server at 127.24.216.190:40503
I20260929 02:13:03.738986 894 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:03.739686 894 heartbeater.cc:511] Master 127.24.216.190:40503 requested a full tablet report, sending...
I20260929 02:13:03.741173 548 ts_manager.cc:194] Registered new tserver with Master: 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635)
I20260929 02:13:03.742414 548 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:55239
I20260929 02:13:03.747273 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:03.725069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:44521 (local address 127.24.216.190:40503)
0929 02:13:03.725256 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:03.725264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:03.725856 (+ 592us) server_negotiation.cc:407] Connection header received
0929 02:13:03.726043 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:03.726050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:03.726135 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:03.726252 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:03.727849 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.729021 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.731486 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.731996 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.732116 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:03.734380 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:03.734385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:03.734388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:03.734415 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:03.737717 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:03.738478 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:03.738482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:03.738484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:03.738607 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:03.739067 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:03.739072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:03.739075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:03.739453 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:03.739598 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:03.746495 (+ 6897us) server_negotiation.cc:299] Negotiation successful
0929 02:13:03.746845 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 02:13:03.748574 874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:03.724720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:40503 (local address 127.24.216.130:44521)
0929 02:13:03.725753 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:03.725771 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:03.725921 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:03.726292 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:03.726299 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:03.727073 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:03.727625 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:03.727641 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.729178 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.729187 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:03.731348 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:03.731361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:03.732224 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:03.732238 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:03.732446 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:03.733215 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:03.733238 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:03.734215 (+ 977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:03.737839 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:03.737845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:03.737849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:03.738369 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:03.738745 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:03.738753 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:03.738759 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:03.738910 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:03.739621 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:03.739631 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:03.739836 (+ 205us) client_negotiation.cc:769] Sending connection context
0929 02:13:03.747729 (+ 7893us) client_negotiation.cc:240] Negotiation successful
0929 02:13:03.748101 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":812,"threads_started":1}
WARNING: no policy specified for oryx/127.24.216.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.131.keytab.
Entry for principal oryx/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.131.keytab.
I20260929 02:13:03.788758 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:40503
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.131.keytab
--principal=oryx/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:13:04.100618 898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:04.100893 898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:04.100982 898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:04.108013 898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:04.108114 898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:04.108192 898 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:13:04.108261 898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:13:04.117159 898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.131.keytab
--principal=oryx/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.216.190:40503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:04.118276 898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:04.119499 898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:04.131567 906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:04.131610 903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:04.131650 904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:04.132432 898 server_base.cc:1034] running on GCE node
I20260929 02:13:04.133040 898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:04.133837 898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:04.135079 898 hybrid_clock.cc:648] HybridClock initialized: now 1790647984135015 us; error 40 us; skew 500 ppm
I20260929 02:13:04.135401 898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:04 dist-test-slave-9k0h krb5kdc[503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647984, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:04.138340 898 init.cc:377] Logged in from keytab as oryx/127.24.216.131@KRBTEST.COM (short username oryx)
I20260929 02:13:04.139706 898 webserver.cc:460] Webserver started at http://127.24.216.131:35859/ using document root <none> and password file <none>
I20260929 02:13:04.140383 898 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:04.140514 898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:04.140837 898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:04.143723 898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "860d55bbf11e4d40a9149be66ef480a1"
format_stamp: "Formatted at 2026-09-29 02:13:04 on dist-test-slave-9k0h"
server_key: "27a587354d25a44ea399523fa36c59ee"
server_key_iv: "e66b7bf379cab0a3b3453f075f673083"
server_key_version: "encryptionkey@0"
I20260929 02:13:04.144500 898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "860d55bbf11e4d40a9149be66ef480a1"
format_stamp: "Formatted at 2026-09-29 02:13:04 on dist-test-slave-9k0h"
server_key: "27a587354d25a44ea399523fa36c59ee"
server_key_iv: "e66b7bf379cab0a3b3453f075f673083"
server_key_version: "encryptionkey@0"
I20260929 02:13:04.149566 898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:13:04.152901 913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:04.154107 898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 02:13:04.154281 898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "860d55bbf11e4d40a9149be66ef480a1"
format_stamp: "Formatted at 2026-09-29 02:13:04 on dist-test-slave-9k0h"
server_key: "27a587354d25a44ea399523fa36c59ee"
server_key_iv: "e66b7bf379cab0a3b3453f075f673083"
server_key_version: "encryptionkey@0"
I20260929 02:13:04.154520 898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:04.168464 898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:04.172188 898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:04.172456 898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:04.175146 898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:04.175276 898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:04.175472 898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:04.175578 898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:13:04 dist-test-slave-9k0h krb5kdc[503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647984, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
I20260929 02:13:04.199249 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.175539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:58239 (local address 127.24.216.190:40503)
0929 02:13:04.175708 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:04.175712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:04.177126 (+ 1414us) server_negotiation.cc:407] Connection header received
0929 02:13:04.179429 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:04.179434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:04.179521 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:04.179633 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:04.181740 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.182798 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.185953 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.186390 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.186519 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.191012 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:04.191021 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:04.191028 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:04.191079 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:04.195095 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:04.195981 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:04.195989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:04.195995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:04.196163 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:04.196817 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:04.196822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:04.196825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:04.197232 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:04.197398 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:04.198402 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0929 02:13:04.198718 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:13:04.199812 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.176140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:40503 (local address 127.24.216.131:58239)
0929 02:13:04.176959 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:04.177000 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:04.179101 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:04.179872 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:04.179880 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:04.180667 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:04.181553 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:04.181583 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.182998 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.183007 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:04.185781 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:04.185809 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.186611 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.186618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:04.186923 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.187632 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:04.187667 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:04.190751 (+ 3084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:04.195241 (+ 4490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:04.195253 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:04.195271 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:04.195847 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:04.196312 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:04.196320 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:04.196325 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:04.196630 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:04.197487 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:04.197498 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:04.197925 (+ 427us) client_negotiation.cc:769] Sending connection context
0929 02:13:04.198264 (+ 339us) client_negotiation.cc:240] Negotiation successful
0929 02:13:04.198735 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":385,"threads_started":1}
I20260929 02:13:04.225291 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.208847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:41507 (local address 127.24.216.190:40503)
0929 02:13:04.209051 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:04.209059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:04.210898 (+ 1839us) server_negotiation.cc:407] Connection header received
0929 02:13:04.211132 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:04.211137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:04.211207 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:04.211309 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:04.212673 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.213630 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.215650 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.216273 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.216344 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.218760 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:04.218765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:04.218770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:04.218809 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:04.222311 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:04.223236 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:04.223239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:04.223241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:04.223375 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:04.223834 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:04.223837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:04.223838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:04.224186 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:04.224320 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:04.224724 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 02:13:04.224927 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:13:04.225440 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.208832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:40503 (local address 127.24.216.131:41507)
0929 02:13:04.210782 (+ 1950us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:04.210826 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:04.210982 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:04.211336 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:04.211340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:04.212056 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:04.212483 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:04.212499 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.213748 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.213753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:04.215471 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:04.215489 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.216636 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.216643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:04.216837 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.217492 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:04.217518 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:04.218593 (+ 1075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:04.222478 (+ 3885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:04.222485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:04.222490 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:04.223082 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:04.223518 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:04.223525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:04.223534 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:04.223695 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:04.224285 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:04.224298 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:04.224478 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 02:13:04.224610 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 02:13:04.224897 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1772,"thread_start_us":1487,"threads_started":1}
I20260929 02:13:04.227291 898 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:39709
I20260929 02:13:04.227320 1032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:39709 every 8 connection(s)
I20260929 02:13:04.228684 898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:13:04.230708 1033 heartbeater.cc:347] Connected to a master server at 127.24.216.190:40503
I20260929 02:13:04.230891 1033 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:04.231431 1033 heartbeater.cc:511] Master 127.24.216.190:40503 requested a full tablet report, sending...
I20260929 02:13:04.232690 548 ts_manager.cc:194] Registered new tserver with Master: 860d55bbf11e4d40a9149be66ef480a1 (127.24.216.131:39709)
I20260929 02:13:04.233412 548 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:58239
I20260929 02:13:04.238695 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 898
I20260929 02:13:04.238895 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:13:04.239419 25442 external_mini_cluster.cc:1428] Setting key 0d8fad1f670f8e6489b37815894673c4
I20260929 02:13:04.242233 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:13:04.249423 755 heartbeater.cc:503] Master 127.24.216.190:40503 was elected leader, sending a full tablet report...
Sep 29 02:13:04 dist-test-slave-9k0h krb5kdc[503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647984, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:13:04 dist-test-slave-9k0h krb5kdc[503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:04 dist-test-slave-9k0h krb5kdc[503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:13:04.291338 643 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:04.281381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.24.216.190:40503)
0929 02:13:04.281622 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:04.281640 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:04.281815 (+ 175us) server_negotiation.cc:407] Connection header received
0929 02:13:04.281958 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:04.281962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:04.282024 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:04.282108 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:04.283424 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.284107 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.285502 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.285915 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.286007 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.290991 (+ 4984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43198: BlockingRecv error: recv got EOF from 127.0.0.1:43198 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 29 02:13:04 dist-test-slave-9k0h krb5kdc[503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647984, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
I20260929 02:13:04.306308 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.294090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.24.216.190:40503)
0929 02:13:04.294305 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:04.294309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:04.294688 (+ 379us) server_negotiation.cc:407] Connection header received
0929 02:13:04.294804 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:04.294807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:04.294870 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:04.294959 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:04.295970 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.296639 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.297767 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.298133 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.298303 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.301609 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:04.301614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:04.301617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:04.301644 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:04.304331 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:04.304868 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:04.304871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:04.304873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:04.304975 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:04.305247 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:04.305250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:04.305252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:04.305494 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:04.305716 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:04.305890 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 02:13:04.306026 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 02:13:04.309801 548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:13:04.316280 548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:13:04.343663 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.327144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.24.216.130:42635)
0929 02:13:04.329846 (+ 2702us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:04.329855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:04.329884 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:13:04.329983 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:04.329991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:04.330279 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:04.332403 (+ 2124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:13:04.332624 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.335428 (+ 2804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.340206 (+ 4778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.341912 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.342957 (+ 1045us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.343150 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:04.343216 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 02:13:04.343392 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2553,"thread_start_us":2425,"threads_started":1}
I20260929 02:13:04.344153 1052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.326985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:42635 (local address 127.0.0.1:49348)
0929 02:13:04.329009 (+ 2024us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:04.329055 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:04.329255 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:04.331639 (+ 2384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:04.331651 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:04.331689 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:13:04.332161 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:04.332175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.335606 (+ 3431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.335615 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:04.339942 (+ 4327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:04.339961 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.342178 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.342186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:04.342442 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.342507 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 02:13:04.342601 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 02:13:04.342784 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1722,"thread_start_us":15197,"threads_started":1}
I20260929 02:13:04.348583 1054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.327601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.24.216.129:46435)
0929 02:13:04.341997 (+ 14396us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:04.342008 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:04.342044 (+ 36us) server_negotiation.cc:407] Connection header received
0929 02:13:04.342155 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:04.342166 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:04.342444 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:04.342640 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:13:04.343228 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.344614 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.346725 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.347584 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.347775 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.347937 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:04.348148 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 02:13:04.348294 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14156,"spinlock_wait_cycles":22912,"thread_start_us":14132,"threads_started":1}
I20260929 02:13:04.348598 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.327085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:46435 (local address 127.0.0.1:47298)
0929 02:13:04.341551 (+ 14466us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:04.341569 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:04.341674 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:04.342661 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:04.342665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:04.342689 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:13:04.343048 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:04.343060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.344755 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.344761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:04.346583 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:04.346594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.347794 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.347800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:04.347945 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.347979 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:13:04.348112 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 02:13:04.348227 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14342,"mutex_wait_us":1,"thread_start_us":1450,"threads_started":1}
I20260929 02:13:04.348043 824 tablet_service.cc:1467] Processing CreateTablet for tablet f9de81f6deea4188ac7b415a2078e874 (DEFAULT_TABLE table=test-table [id=b1e6d33104c34bba9b095741362416d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:04.349483 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9de81f6deea4188ac7b415a2078e874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:04.352233 685 tablet_service.cc:1467] Processing CreateTablet for tablet f9de81f6deea4188ac7b415a2078e874 (DEFAULT_TABLE table=test-table [id=b1e6d33104c34bba9b095741362416d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:04.353508 685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9de81f6deea4188ac7b415a2078e874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:04.353897 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.329439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:39709 (local address 127.0.0.1:43198)
0929 02:13:04.342451 (+ 13012us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:04.342466 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:04.342577 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:04.343432 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:04.343436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:04.343460 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:13:04.343864 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:04.343878 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.345948 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.345954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:04.348125 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:04.348140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.353125 (+ 4985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.353130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:04.353286 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.353319 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:13:04.353431 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 02:13:04.353545 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12899,"thread_start_us":11862,"threads_started":1}
I20260929 02:13:04.353950 1056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.330066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.24.216.131:39709)
0929 02:13:04.335710 (+ 5644us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:04.335721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:04.342572 (+ 6851us) server_negotiation.cc:407] Connection header received
0929 02:13:04.342754 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:04.342762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:04.342984 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:04.343150 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:13:04.344085 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.345811 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.348303 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.352928 (+ 4625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.353126 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.353282 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:04.353468 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 02:13:04.353627 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5438,"thread_start_us":5214,"threads_started":1}
I20260929 02:13:04.357162 963 tablet_service.cc:1467] Processing CreateTablet for tablet f9de81f6deea4188ac7b415a2078e874 (DEFAULT_TABLE table=test-table [id=b1e6d33104c34bba9b095741362416d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:04.358415 963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9de81f6deea4188ac7b415a2078e874. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:04.363770 1057 tablet_bootstrap.cc:492] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45: Bootstrap starting.
I20260929 02:13:04.367121 1058 tablet_bootstrap.cc:492] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88: Bootstrap starting.
I20260929 02:13:04.368268 1057 tablet_bootstrap.cc:654] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:04.369593 1057 log.cc:824] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45: Log is configured to *not* fsync() on all Append() calls
I20260929 02:13:04.370128 1058 tablet_bootstrap.cc:654] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:04.370357 1059 tablet_bootstrap.cc:492] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1: Bootstrap starting.
I20260929 02:13:04.371433 1058 log.cc:824] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88: Log is configured to *not* fsync() on all Append() calls
I20260929 02:13:04.373113 1057 tablet_bootstrap.cc:492] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45: No bootstrap required, opened a new log
I20260929 02:13:04.373363 1057 ts_tablet_manager.cc:1397] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260929 02:13:04.373558 1059 tablet_bootstrap.cc:654] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:04.374415 1058 tablet_bootstrap.cc:492] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88: No bootstrap required, opened a new log
I20260929 02:13:04.374666 1058 ts_tablet_manager.cc:1397] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260929 02:13:04.374739 1059 log.cc:824] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:13:04.377693 1059 tablet_bootstrap.cc:492] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1: No bootstrap required, opened a new log
I20260929 02:13:04.377945 1059 ts_tablet_manager.cc:1397] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 02:13:04.381093 1057 raft_consensus.cc:348] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.381434 1057 raft_consensus.cc:374] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:04.381531 1057 raft_consensus.cc:729] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4133caddbdd84b8dbc4ea2f3fff2de45, State: Initialized, Role: FOLLOWER
I20260929 02:13:04.382133 1057 consensus_queue.cc:260] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.384740 894 heartbeater.cc:503] Master 127.24.216.190:40503 was elected leader, sending a full tablet report...
I20260929 02:13:04.385480 1057 ts_tablet_manager.cc:1428] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45: Time spent starting tablet: real 0.012s user 0.006s sys 0.007s
I20260929 02:13:04.385458 1058 raft_consensus.cc:348] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.385944 1058 raft_consensus.cc:374] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:04.386041 1058 raft_consensus.cc:729] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6faaba97bf4b9691d4c6536df17c88, State: Initialized, Role: FOLLOWER
I20260929 02:13:04.386642 1058 consensus_queue.cc:260] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.388149 1058 ts_tablet_manager.cc:1428] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260929 02:13:04.387964 1059 raft_consensus.cc:348] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.388370 1059 raft_consensus.cc:374] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:04.388494 1059 raft_consensus.cc:729] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860d55bbf11e4d40a9149be66ef480a1, State: Initialized, Role: FOLLOWER
I20260929 02:13:04.389323 1059 consensus_queue.cc:260] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.390774 1033 heartbeater.cc:503] Master 127.24.216.190:40503 was elected leader, sending a full tablet report...
I20260929 02:13:04.392165 1059 ts_tablet_manager.cc:1428] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260929 02:13:04.435108 1064 raft_consensus.cc:482] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:13:04.435433 1064 raft_consensus.cc:504] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.436985 1064 leader_election.cc:288] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635), 860d55bbf11e4d40a9149be66ef480a1 (127.24.216.131:39709)
I20260929 02:13:04.443625 1056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.437517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:51281 (local address 127.24.216.131:39709)
0929 02:13:04.437819 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:04.437825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:04.438346 (+ 521us) server_negotiation.cc:407] Connection header received
0929 02:13:04.438525 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:04.438529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:04.438582 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:04.438673 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:13:04.439280 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.439987 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.441845 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.442615 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.442787 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.442873 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:04.443206 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 02:13:04.443329 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260929 02:13:04.443632 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.437402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:40315 (local address 127.24.216.130:42635)
0929 02:13:04.437631 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:04.437640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:04.438287 (+ 647us) server_negotiation.cc:407] Connection header received
0929 02:13:04.438513 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:04.438517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:04.438571 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:04.438661 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:13:04.439221 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.439975 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.441881 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.442649 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.442833 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.442908 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:04.443224 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 02:13:04.443345 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:13:04.443763 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.437269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:42635 (local address 127.24.216.129:40315)
0929 02:13:04.438142 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:04.438191 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:04.438352 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:04.438675 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:04.438679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:04.438701 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:13:04.439078 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:04.439089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.440098 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.440106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:04.441707 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:04.441718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.442872 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.442878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:04.443078 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.443100 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:13:04.443201 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 02:13:04.443354 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":44,"thread_start_us":493,"threads_started":1}
I20260929 02:13:04.444087 1067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.437370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:39709 (local address 127.24.216.129:51281)
0929 02:13:04.438265 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:04.438278 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:04.438382 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:04.438695 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:04.438698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:04.438720 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:13:04.439120 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:04.439128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.440119 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.440125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:04.441687 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:04.441701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.442831 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.442837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:04.443041 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.443077 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 02:13:04.443175 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 02:13:04.443354 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":126,"thread_start_us":607,"threads_started":1}
I20260929 02:13:04.444787 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9de81f6deea4188ac7b415a2078e874" candidate_uuid: "ca6faaba97bf4b9691d4c6536df17c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" is_pre_election: true
I20260929 02:13:04.444990 988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9de81f6deea4188ac7b415a2078e874" candidate_uuid: "ca6faaba97bf4b9691d4c6536df17c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "860d55bbf11e4d40a9149be66ef480a1" is_pre_election: true
I20260929 02:13:04.445273 844 raft_consensus.cc:2454] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca6faaba97bf4b9691d4c6536df17c88 in term 0.
I20260929 02:13:04.445518 988 raft_consensus.cc:2454] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca6faaba97bf4b9691d4c6536df17c88 in term 0.
I20260929 02:13:04.445953 637 leader_election.cc:302] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4133caddbdd84b8dbc4ea2f3fff2de45, ca6faaba97bf4b9691d4c6536df17c88; no voters:
I20260929 02:13:04.446403 1064 raft_consensus.cc:2781] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:13:04.446543 1064 raft_consensus.cc:482] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:13:04.446640 1064 raft_consensus.cc:3037] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:04.449031 1064 raft_consensus.cc:504] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.449608 1064 leader_election.cc:288] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [CANDIDATE]: Term 1 election: Requested vote from peers 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635), 860d55bbf11e4d40a9149be66ef480a1 (127.24.216.131:39709)
I20260929 02:13:04.449990 988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9de81f6deea4188ac7b415a2078e874" candidate_uuid: "ca6faaba97bf4b9691d4c6536df17c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "860d55bbf11e4d40a9149be66ef480a1"
I20260929 02:13:04.449994 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9de81f6deea4188ac7b415a2078e874" candidate_uuid: "ca6faaba97bf4b9691d4c6536df17c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45"
I20260929 02:13:04.450153 988 raft_consensus.cc:3037] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:04.450199 844 raft_consensus.cc:3037] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:04.452253 988 raft_consensus.cc:2454] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca6faaba97bf4b9691d4c6536df17c88 in term 1.
I20260929 02:13:04.452275 844 raft_consensus.cc:2454] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca6faaba97bf4b9691d4c6536df17c88 in term 1.
I20260929 02:13:04.452651 635 leader_election.cc:302] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 860d55bbf11e4d40a9149be66ef480a1, ca6faaba97bf4b9691d4c6536df17c88; no voters:
I20260929 02:13:04.452939 1064 raft_consensus.cc:2781] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:13:04.453374 1064 raft_consensus.cc:686] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 LEADER]: Becoming Leader. State: Replica: ca6faaba97bf4b9691d4c6536df17c88, State: Running, Role: LEADER
I20260929 02:13:04.453795 1064 consensus_queue.cc:237] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.457930 547 catalog_manager.cc:5495] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca6faaba97bf4b9691d4c6536df17c88 (127.24.216.129). New cstate: current_term: 1 leader_uuid: "ca6faaba97bf4b9691d4c6536df17c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } health_report { overall_health: UNKNOWN } } }
W20260929 02:13:04.480360 1034 tablet.cc:2369] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:13:04.487377 895 tablet.cc:2369] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:13:04.490017 1054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:04.484410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.24.216.129:46435)
0929 02:13:04.484604 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:04.484611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:04.484700 (+ 89us) server_negotiation.cc:407] Connection header received
0929 02:13:04.484959 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:04.484964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:04.485058 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:04.485197 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:13:04.485553 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.486408 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.488085 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:04.488469 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:04.488569 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:04.488982 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:13:04.489463 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:13:04.489567 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:04.489667 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 02:13:04.489727 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 02:13:04.493147 756 tablet.cc:2369] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:13:04.497816 988 raft_consensus.cc:1261] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Refusing update from remote peer ca6faaba97bf4b9691d4c6536df17c88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:13:04.497800 844 raft_consensus.cc:1261] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Refusing update from remote peer ca6faaba97bf4b9691d4c6536df17c88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:13:04.498814 1068 consensus_queue.cc:1035] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:13:04.499127 1064 consensus_queue.cc:1035] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:13:04.517050 1071 mvcc.cc:205] Tried to move back new op lower bound from 7334494144493977600 to 7334494144326389760. Current Snapshot: MvccSnapshot[applied={T|T < 7334494144493977600 or (T in {7334494144493977600})}]
I20260929 02:13:04.526106 547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:13:04.526414 547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:13:04.529708 547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7: Sending DeleteTablet for 3 replicas of tablet f9de81f6deea4188ac7b415a2078e874
I20260929 02:13:04.530722 824 tablet_service.cc:1514] Processing DeleteTablet for tablet f9de81f6deea4188ac7b415a2078e874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:04 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:49348
I20260929 02:13:04.530835 685 tablet_service.cc:1514] Processing DeleteTablet for tablet f9de81f6deea4188ac7b415a2078e874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:04 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:47298
I20260929 02:13:04.531066 963 tablet_service.cc:1514] Processing DeleteTablet for tablet f9de81f6deea4188ac7b415a2078e874 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:04 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:43198
I20260929 02:13:04.533847 547 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:13:04.535533 547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:13:04.541812 1081 tablet_replica.cc:330] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88: stopping tablet replica
I20260929 02:13:04.542598 1081 raft_consensus.cc:2229] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:13:04.543497 1081 raft_consensus.cc:2258] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:04.545354 823 tablet_service.cc:1467] Processing CreateTablet for tablet d3fddd068c67430da75b67741a9a6415 (DEFAULT_TABLE table=test-table [id=b8847fb9843f4c91955ae4f936e5b0c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:04.545522 962 tablet_service.cc:1467] Processing CreateTablet for tablet d3fddd068c67430da75b67741a9a6415 (DEFAULT_TABLE table=test-table [id=b8847fb9843f4c91955ae4f936e5b0c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:04.546003 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3fddd068c67430da75b67741a9a6415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:04.546180 962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3fddd068c67430da75b67741a9a6415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:04.551613 1057 tablet_bootstrap.cc:492] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45: Bootstrap starting.
I20260929 02:13:04.553258 1057 tablet_bootstrap.cc:654] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:04.556139 1059 tablet_bootstrap.cc:492] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1: Bootstrap starting.
I20260929 02:13:04.556792 1082 tablet_replica.cc:330] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1: stopping tablet replica
I20260929 02:13:04.557415 1080 tablet_replica.cc:330] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45: stopping tablet replica
I20260929 02:13:04.550750 685 tablet_service.cc:1467] Processing CreateTablet for tablet d3fddd068c67430da75b67741a9a6415 (DEFAULT_TABLE table=test-table [id=b8847fb9843f4c91955ae4f936e5b0c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:04.557845 1059 tablet_bootstrap.cc:654] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:04.558001 685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3fddd068c67430da75b67741a9a6415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:04.566879 1080 raft_consensus.cc:2229] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:13:04.567148 1082 raft_consensus.cc:2229] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:13:04.567828 1082 raft_consensus.cc:2258] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:04.568375 1080 raft_consensus.cc:2258] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:04.568156 1059 tablet_bootstrap.cc:492] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1: No bootstrap required, opened a new log
I20260929 02:13:04.568668 1059 ts_tablet_manager.cc:1397] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.011s
I20260929 02:13:04.569023 1081 ts_tablet_manager.cc:1905] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:04.569072 1058 tablet_bootstrap.cc:492] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88: Bootstrap starting.
I20260929 02:13:04.569821 1059 raft_consensus.cc:348] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.570065 1059 raft_consensus.cc:374] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:04.570190 1059 raft_consensus.cc:729] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860d55bbf11e4d40a9149be66ef480a1, State: Initialized, Role: FOLLOWER
I20260929 02:13:04.570480 1059 consensus_queue.cc:260] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.570804 1058 tablet_bootstrap.cc:654] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:04.571455 1057 tablet_bootstrap.cc:492] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45: No bootstrap required, opened a new log
I20260929 02:13:04.571584 1057 ts_tablet_manager.cc:1397] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.005s
I20260929 02:13:04.572376 1057 raft_consensus.cc:348] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.572600 1057 raft_consensus.cc:374] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:04.572693 1057 raft_consensus.cc:729] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4133caddbdd84b8dbc4ea2f3fff2de45, State: Initialized, Role: FOLLOWER
I20260929 02:13:04.574105 1059 ts_tablet_manager.cc:1428] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260929 02:13:04.574810 1082 ts_tablet_manager.cc:1905] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:04.575356 1057 consensus_queue.cc:260] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.575928 1057 ts_tablet_manager.cc:1428] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 02:13:04.577769 1080 ts_tablet_manager.cc:1905] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:04.580984 1081 ts_tablet_manager.cc:1918] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:13:04.581168 1081 log.cc:1196] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/f9de81f6deea4188ac7b415a2078e874
I20260929 02:13:04.581642 1081 ts_tablet_manager.cc:1939] T f9de81f6deea4188ac7b415a2078e874 P ca6faaba97bf4b9691d4c6536df17c88: Deleting consensus metadata
I20260929 02:13:04.583518 532 catalog_manager.cc:4841] TS ca6faaba97bf4b9691d4c6536df17c88 (127.24.216.129:46435): tablet f9de81f6deea4188ac7b415a2078e874 (table test-table [id=b1e6d33104c34bba9b095741362416d4]) successfully deleted
I20260929 02:13:04.583602 1058 tablet_bootstrap.cc:492] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88: No bootstrap required, opened a new log
I20260929 02:13:04.583727 1058 ts_tablet_manager.cc:1397] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260929 02:13:04.584565 1058 raft_consensus.cc:348] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.584774 1058 raft_consensus.cc:374] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:04.584896 1058 raft_consensus.cc:729] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6faaba97bf4b9691d4c6536df17c88, State: Initialized, Role: FOLLOWER
I20260929 02:13:04.585170 1058 consensus_queue.cc:260] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.586411 1058 ts_tablet_manager.cc:1428] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:13:04.586941 1080 ts_tablet_manager.cc:1918] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:13:04.587083 1080 log.cc:1196] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/f9de81f6deea4188ac7b415a2078e874
I20260929 02:13:04.587644 1080 ts_tablet_manager.cc:1939] T f9de81f6deea4188ac7b415a2078e874 P 4133caddbdd84b8dbc4ea2f3fff2de45: Deleting consensus metadata
I20260929 02:13:04.588485 1082 ts_tablet_manager.cc:1918] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:13:04.588629 1082 log.cc:1196] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/f9de81f6deea4188ac7b415a2078e874
I20260929 02:13:04.589133 1082 ts_tablet_manager.cc:1939] T f9de81f6deea4188ac7b415a2078e874 P 860d55bbf11e4d40a9149be66ef480a1: Deleting consensus metadata
I20260929 02:13:04.589368 534 catalog_manager.cc:4841] TS 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635): tablet f9de81f6deea4188ac7b415a2078e874 (table test-table [id=b1e6d33104c34bba9b095741362416d4]) successfully deleted
I20260929 02:13:04.590716 532 catalog_manager.cc:4841] TS 860d55bbf11e4d40a9149be66ef480a1 (127.24.216.131:39709): tablet f9de81f6deea4188ac7b415a2078e874 (table test-table [id=b1e6d33104c34bba9b095741362416d4]) successfully deleted
I20260929 02:13:04.906329 1068 raft_consensus.cc:482] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:13:04.906589 1068 raft_consensus.cc:504] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.907262 1068 leader_election.cc:288] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635), 860d55bbf11e4d40a9149be66ef480a1 (127.24.216.131:39709)
I20260929 02:13:04.907821 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fddd068c67430da75b67741a9a6415" candidate_uuid: "ca6faaba97bf4b9691d4c6536df17c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" is_pre_election: true
I20260929 02:13:04.907819 988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fddd068c67430da75b67741a9a6415" candidate_uuid: "ca6faaba97bf4b9691d4c6536df17c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "860d55bbf11e4d40a9149be66ef480a1" is_pre_election: true
I20260929 02:13:04.908062 988 raft_consensus.cc:2454] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca6faaba97bf4b9691d4c6536df17c88 in term 0.
I20260929 02:13:04.908061 844 raft_consensus.cc:2454] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca6faaba97bf4b9691d4c6536df17c88 in term 0.
I20260929 02:13:04.908470 637 leader_election.cc:302] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4133caddbdd84b8dbc4ea2f3fff2de45, ca6faaba97bf4b9691d4c6536df17c88; no voters:
I20260929 02:13:04.908808 1068 raft_consensus.cc:2781] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:13:04.908897 1068 raft_consensus.cc:482] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:13:04.909045 1068 raft_consensus.cc:3037] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:04.911269 1068 raft_consensus.cc:504] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.911859 1068 leader_election.cc:288] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [CANDIDATE]: Term 1 election: Requested vote from peers 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635), 860d55bbf11e4d40a9149be66ef480a1 (127.24.216.131:39709)
I20260929 02:13:04.912258 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fddd068c67430da75b67741a9a6415" candidate_uuid: "ca6faaba97bf4b9691d4c6536df17c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45"
I20260929 02:13:04.912315 988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fddd068c67430da75b67741a9a6415" candidate_uuid: "ca6faaba97bf4b9691d4c6536df17c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "860d55bbf11e4d40a9149be66ef480a1"
I20260929 02:13:04.912461 844 raft_consensus.cc:3037] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:04.912482 988 raft_consensus.cc:3037] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:04.915222 844 raft_consensus.cc:2454] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca6faaba97bf4b9691d4c6536df17c88 in term 1.
I20260929 02:13:04.915221 988 raft_consensus.cc:2454] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca6faaba97bf4b9691d4c6536df17c88 in term 1.
I20260929 02:13:04.915597 635 leader_election.cc:302] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 860d55bbf11e4d40a9149be66ef480a1, ca6faaba97bf4b9691d4c6536df17c88; no voters:
I20260929 02:13:04.915895 1068 raft_consensus.cc:2781] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:13:04.916054 1068 raft_consensus.cc:686] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 LEADER]: Becoming Leader. State: Replica: ca6faaba97bf4b9691d4c6536df17c88, State: Running, Role: LEADER
I20260929 02:13:04.916297 1068 consensus_queue.cc:237] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:04.918980 547 catalog_manager.cc:5495] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca6faaba97bf4b9691d4c6536df17c88 (127.24.216.129). New cstate: current_term: 1 leader_uuid: "ca6faaba97bf4b9691d4c6536df17c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } health_report { overall_health: UNKNOWN } } }
Sep 29 02:13:05 dist-test-slave-9k0h krb5kdc[503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647982, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.190@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
I20260929 02:13:05.053220 1092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.036343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43214 (local address 127.24.216.190:40503)
0929 02:13:05.037043 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:05.037049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:05.037199 (+ 150us) server_negotiation.cc:407] Connection header received
0929 02:13:05.037393 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:05.037398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:05.037467 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:05.037571 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:05.038663 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.039435 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.040635 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.041034 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.041167 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.048240 (+ 7073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:05.048244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:05.048247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:05.048272 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:05.050696 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:05.051359 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:05.051370 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:05.051373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:05.051479 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:05.051829 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:05.051833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:05.051837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:05.052077 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:05.052217 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:05.052578 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 02:13:05.052869 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":178,"server-negotiator.queue_time_us":529,"thread_start_us":348,"threads_started":1}
I20260929 02:13:05.053241 1093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.036356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:40503 (local address 127.0.0.1:43214)
0929 02:13:05.037107 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:05.037122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:05.037274 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:05.037605 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:05.037611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:05.038342 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:05.038533 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:05.038542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.039574 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.039579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:05.040502 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:05.040512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.041180 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.041185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:05.041396 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.045888 (+ 4492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:05.045903 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:05.048062 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:05.050838 (+ 2776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:05.050848 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:05.050856 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:05.051236 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:05.051590 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:05.051597 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:05.051600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:05.051721 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:05.052193 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:05.052201 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:05.052359 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 02:13:05.052510 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 02:13:05.052775 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":23,"thread_start_us":393,"threads_started":1}
I20260929 02:13:05.057997 547 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:43214:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:13:05.059068 547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:13:05.064904 685 tablet_service.cc:1467] Processing CreateTablet for tablet e0e70255ccfa464f921fdf7f204d042c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=075e77d1502542839ad9d2045eb04a32]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:13:05.065099 962 tablet_service.cc:1467] Processing CreateTablet for tablet e0e70255ccfa464f921fdf7f204d042c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=075e77d1502542839ad9d2045eb04a32]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:13:05.065106 823 tablet_service.cc:1467] Processing CreateTablet for tablet e0e70255ccfa464f921fdf7f204d042c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=075e77d1502542839ad9d2045eb04a32]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:13:05.065500 685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e70255ccfa464f921fdf7f204d042c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:05.065599 962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e70255ccfa464f921fdf7f204d042c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:05.065599 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e70255ccfa464f921fdf7f204d042c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:05.070905 1058 tablet_bootstrap.cc:492] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88: Bootstrap starting.
I20260929 02:13:05.072127 1058 tablet_bootstrap.cc:654] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:05.072532 1057 tablet_bootstrap.cc:492] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45: Bootstrap starting.
I20260929 02:13:05.072532 1059 tablet_bootstrap.cc:492] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1: Bootstrap starting.
I20260929 02:13:05.074014 1059 tablet_bootstrap.cc:654] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:05.074014 1057 tablet_bootstrap.cc:654] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:05.074122 1058 tablet_bootstrap.cc:492] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88: No bootstrap required, opened a new log
I20260929 02:13:05.074301 1058 ts_tablet_manager.cc:1397] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260929 02:13:05.074904 1058 raft_consensus.cc:348] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.075157 1058 raft_consensus.cc:374] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:05.075246 1058 raft_consensus.cc:729] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6faaba97bf4b9691d4c6536df17c88, State: Initialized, Role: FOLLOWER
I20260929 02:13:05.075639 1058 consensus_queue.cc:260] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.076053 1068 tablet_replica.cc:439] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } } }
I20260929 02:13:05.076239 1059 tablet_bootstrap.cc:492] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1: No bootstrap required, opened a new log
I20260929 02:13:05.076275 1068 tablet_replica.cc:442] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:13:05.076345 1059 ts_tablet_manager.cc:1397] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:13:05.076417 1057 tablet_bootstrap.cc:492] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45: No bootstrap required, opened a new log
I20260929 02:13:05.076550 1057 ts_tablet_manager.cc:1397] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:13:05.076890 1059 raft_consensus.cc:348] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.077129 1059 raft_consensus.cc:374] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:05.077204 1059 raft_consensus.cc:729] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860d55bbf11e4d40a9149be66ef480a1, State: Initialized, Role: FOLLOWER
I20260929 02:13:05.077239 1058 ts_tablet_manager.cc:1428] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 02:13:05.077183 1057 raft_consensus.cc:348] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.077412 1057 raft_consensus.cc:374] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:05.077485 1057 raft_consensus.cc:729] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4133caddbdd84b8dbc4ea2f3fff2de45, State: Initialized, Role: FOLLOWER
I20260929 02:13:05.077415 1059 consensus_queue.cc:260] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.077739 1057 consensus_queue.cc:260] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.078094 1097 tablet_replica.cc:439] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } } }
I20260929 02:13:05.078928 1097 tablet_replica.cc:442] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:13:05.079164 1059 ts_tablet_manager.cc:1428] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:13:05.079712 1098 tablet_replica.cc:439] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } } }
I20260929 02:13:05.079972 1098 tablet_replica.cc:442] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:13:05.081553 1057 ts_tablet_manager.cc:1428] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260929 02:13:05.235003 1097 raft_consensus.cc:482] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:13:05.235208 1097 raft_consensus.cc:504] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.236853 1097 leader_election.cc:288] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca6faaba97bf4b9691d4c6536df17c88 (127.24.216.129:46435), 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635)
I20260929 02:13:05.262413 1099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.237047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:42635 (local address 127.24.216.131:33861)
0929 02:13:05.255127 (+ 18080us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:05.255147 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:05.255311 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:05.255879 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:05.255885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:05.255911 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:13:05.256242 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:05.256256 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.257703 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.257709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:05.259680 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:05.259694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.261552 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.261558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:05.261730 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.261762 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:13:05.261850 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 02:13:05.261975 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17951,"thread_start_us":17819,"threads_started":1}
I20260929 02:13:05.263106 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.237358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:33861 (local address 127.24.216.130:42635)
0929 02:13:05.255503 (+ 18145us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:05.255511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:05.255535 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:13:05.255621 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:05.255628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:05.255744 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:05.255854 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:13:05.256441 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.257554 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.259877 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.261087 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.261229 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.261318 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:05.262629 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0929 02:13:05.262770 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17964,"thread_start_us":18046,"threads_started":1}
I20260929 02:13:05.263541 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.237047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:46435 (local address 127.24.216.131:49625)
0929 02:13:05.256429 (+ 19382us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:05.256444 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:05.256538 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:05.257039 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:05.257043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:05.257064 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:13:05.257426 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:05.257435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.258888 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.258893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:05.260929 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:05.260942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.262726 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.262732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:05.262933 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.262966 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:13:05.263046 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 02:13:05.263177 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19270,"mutex_wait_us":492,"thread_start_us":23753,"threads_started":1}
I20260929 02:13:05.263676 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e70255ccfa464f921fdf7f204d042c" candidate_uuid: "860d55bbf11e4d40a9149be66ef480a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" is_pre_election: true
I20260929 02:13:05.263994 844 raft_consensus.cc:2454] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 860d55bbf11e4d40a9149be66ef480a1 in term 0.
I20260929 02:13:05.264256 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.237309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:49625 (local address 127.24.216.129:46435)
0929 02:13:05.251921 (+ 14612us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:05.251929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:05.256629 (+ 4700us) server_negotiation.cc:407] Connection header received
0929 02:13:05.256695 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:05.256699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:05.256765 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:05.256861 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:13:05.257615 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.258707 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.261093 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.262303 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.262455 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.263706 (+ 1251us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:05.263758 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 02:13:05.263931 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14482,"thread_start_us":14628,"threads_started":1}
I20260929 02:13:05.264523 916 leader_election.cc:302] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4133caddbdd84b8dbc4ea2f3fff2de45, 860d55bbf11e4d40a9149be66ef480a1; no voters:
I20260929 02:13:05.264940 1097 raft_consensus.cc:2781] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:13:05.265070 1097 raft_consensus.cc:482] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:13:05.265177 1097 raft_consensus.cc:3037] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:05.265125 710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e70255ccfa464f921fdf7f204d042c" candidate_uuid: "860d55bbf11e4d40a9149be66ef480a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca6faaba97bf4b9691d4c6536df17c88" is_pre_election: true
I20260929 02:13:05.265497 710 raft_consensus.cc:2454] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 860d55bbf11e4d40a9149be66ef480a1 in term 0.
I20260929 02:13:05.267503 1097 raft_consensus.cc:504] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.268090 1097 leader_election.cc:288] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [CANDIDATE]: Term 1 election: Requested vote from peers ca6faaba97bf4b9691d4c6536df17c88 (127.24.216.129:46435), 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635)
I20260929 02:13:05.268433 710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e70255ccfa464f921fdf7f204d042c" candidate_uuid: "860d55bbf11e4d40a9149be66ef480a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca6faaba97bf4b9691d4c6536df17c88"
I20260929 02:13:05.268625 710 raft_consensus.cc:3037] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:05.268568 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e70255ccfa464f921fdf7f204d042c" candidate_uuid: "860d55bbf11e4d40a9149be66ef480a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45"
I20260929 02:13:05.268770 844 raft_consensus.cc:3037] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:05.271654 844 raft_consensus.cc:2454] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 860d55bbf11e4d40a9149be66ef480a1 in term 1.
I20260929 02:13:05.271730 710 raft_consensus.cc:2454] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 860d55bbf11e4d40a9149be66ef480a1 in term 1.
I20260929 02:13:05.272064 916 leader_election.cc:302] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4133caddbdd84b8dbc4ea2f3fff2de45, 860d55bbf11e4d40a9149be66ef480a1; no voters:
I20260929 02:13:05.272346 1097 raft_consensus.cc:2781] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:13:05.272835 1097 raft_consensus.cc:686] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [term 1 LEADER]: Becoming Leader. State: Replica: 860d55bbf11e4d40a9149be66ef480a1, State: Running, Role: LEADER
I20260929 02:13:05.273193 1097 consensus_queue.cc:237] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.274998 1103 tablet_replica.cc:439] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1: TxnStatusTablet state changed. Reason: New leader 860d55bbf11e4d40a9149be66ef480a1. Latest consensus state: current_term: 1 leader_uuid: "860d55bbf11e4d40a9149be66ef480a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } } }
I20260929 02:13:05.275444 1103 tablet_replica.cc:442] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:13:05.276227 1105 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:13:05.276445 1105 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:13:05.276865 545 catalog_manager.cc:5495] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 860d55bbf11e4d40a9149be66ef480a1 (127.24.216.131). New cstate: current_term: 1 leader_uuid: "860d55bbf11e4d40a9149be66ef480a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } health_report { overall_health: HEALTHY } } }
I20260929 02:13:05.315380 1093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.308752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:39709 (local address 127.0.0.1:43206)
0929 02:13:05.309010 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:05.309034 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:05.309199 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:05.309912 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:05.309915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:05.309924 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:13:05.310072 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:05.310079 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.311728 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.311731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:05.312692 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:05.312702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.313676 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.313680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:05.313818 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.313866 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:13:05.315004 (+ 1138us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:13:05.315018 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 02:13:05.315072 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 02:13:05.315175 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 02:13:05.315583 1108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.308808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.24.216.131:39709)
0929 02:13:05.309491 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:05.309497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:05.309523 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:13:05.309619 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:05.309624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:05.309725 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:05.309888 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:13:05.310268 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.311616 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.312827 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.313540 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.313677 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.314076 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:13:05.314903 (+ 827us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:13:05.315052 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:05.315107 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 02:13:05.315178 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":362,"threads_started":1}
I20260929 02:13:05.319541 844 raft_consensus.cc:1261] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Refusing update from remote peer 860d55bbf11e4d40a9149be66ef480a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:13:05.319835 710 raft_consensus.cc:1261] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Refusing update from remote peer 860d55bbf11e4d40a9149be66ef480a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:13:05.320241 1103 consensus_queue.cc:1035] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:13:05.320575 1097 consensus_queue.cc:1035] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:13:05.323983 1098 tablet_replica.cc:439] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45: TxnStatusTablet state changed. Reason: New leader 860d55bbf11e4d40a9149be66ef480a1. Latest consensus state: current_term: 1 leader_uuid: "860d55bbf11e4d40a9149be66ef480a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } } }
I20260929 02:13:05.324254 1098 tablet_replica.cc:442] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:13:05.325735 1109 mvcc.cc:205] Tried to move back new op lower bound from 7334494147862413312 to 7334494147682217984. Current Snapshot: MvccSnapshot[applied={T|T < 7334494147862413312}]
I20260929 02:13:05.326862 1103 tablet_replica.cc:439] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "860d55bbf11e4d40a9149be66ef480a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } } }
I20260929 02:13:05.327091 1103 tablet_replica.cc:442] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:13:05.327358 1110 mvcc.cc:205] Tried to move back new op lower bound from 7334494147862413312 to 7334494147682217984. Current Snapshot: MvccSnapshot[applied={T|T < 7334494147862413312}]
I20260929 02:13:05.335260 1098 tablet_replica.cc:439] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "860d55bbf11e4d40a9149be66ef480a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } } }
I20260929 02:13:05.335485 1098 tablet_replica.cc:442] T e0e70255ccfa464f921fdf7f204d042c P 4133caddbdd84b8dbc4ea2f3fff2de45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:13:05.343232 1068 tablet_replica.cc:439] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88: TxnStatusTablet state changed. Reason: New leader 860d55bbf11e4d40a9149be66ef480a1. Latest consensus state: current_term: 1 leader_uuid: "860d55bbf11e4d40a9149be66ef480a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } } }
I20260929 02:13:05.343492 1068 tablet_replica.cc:442] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:13:05.343551 1097 tablet_replica.cc:439] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "860d55bbf11e4d40a9149be66ef480a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } } }
I20260929 02:13:05.343811 1097 tablet_replica.cc:442] T e0e70255ccfa464f921fdf7f204d042c P 860d55bbf11e4d40a9149be66ef480a1: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:13:05.349099 1068 tablet_replica.cc:439] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "860d55bbf11e4d40a9149be66ef480a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } } }
I20260929 02:13:05.349356 1068 tablet_replica.cc:442] T e0e70255ccfa464f921fdf7f204d042c P ca6faaba97bf4b9691d4c6536df17c88: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:13:05.355207 1108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.349684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:41425 (local address 127.24.216.131:39709)
0929 02:13:05.349992 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:05.349998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:05.350520 (+ 522us) server_negotiation.cc:407] Connection header received
0929 02:13:05.350811 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:05.350816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:05.350899 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:05.351009 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:13:05.351556 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.352263 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.353499 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.353924 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.354053 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.354515 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:13:05.354608 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:13:05.354706 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:05.354851 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 02:13:05.354921 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260929 02:13:05.355285 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.349269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:39709 (local address 127.24.216.129:41425)
0929 02:13:05.350432 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:05.350450 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:05.350636 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:05.351036 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:05.351042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:05.351055 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:13:05.351413 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:05.351424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.352400 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.352405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:05.353359 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:05.353369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.354067 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.354072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:05.354268 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.354341 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:13:05.354717 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:13:05.354730 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 02:13:05.354824 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 02:13:05.354948 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"thread_start_us":882,"threads_started":1}
I20260929 02:13:05.363441 844 raft_consensus.cc:1261] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Refusing update from remote peer ca6faaba97bf4b9691d4c6536df17c88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:13:05.363444 988 raft_consensus.cc:1261] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Refusing update from remote peer ca6faaba97bf4b9691d4c6536df17c88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:13:05.363992 1068 consensus_queue.cc:1035] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:13:05.364284 1064 consensus_queue.cc:1035] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:13:05.396544 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.390173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:54121 (local address 127.24.216.129:46435)
0929 02:13:05.390355 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:05.390362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:05.391038 (+ 676us) server_negotiation.cc:407] Connection header received
0929 02:13:05.391270 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:05.391275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:05.391361 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:05.391462 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:13:05.392109 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.392979 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.394523 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.395074 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.395202 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.395812 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:13:05.395936 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:13:05.396016 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:05.396171 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 02:13:05.396241 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:13:05.396695 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.390115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:46435 (local address 127.24.216.131:54121)
0929 02:13:05.390926 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:05.390945 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:05.391108 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:05.391476 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:05.391480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:05.391491 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:13:05.391947 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:05.391962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.393131 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.393138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:05.394368 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:05.394379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.395380 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.395387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:05.395587 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.395637 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:13:05.396064 (+ 427us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:13:05.396075 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 02:13:05.396152 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 02:13:05.396274 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":430,"threads_started":1}
I20260929 02:13:05.431620 546 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:13:05.431949 546 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:13:05.434785 546 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7: Sending DeleteTablet for 3 replicas of tablet d3fddd068c67430da75b67741a9a6415
I20260929 02:13:05.435555 823 tablet_service.cc:1514] Processing DeleteTablet for tablet d3fddd068c67430da75b67741a9a6415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:05 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:49348
I20260929 02:13:05.435694 685 tablet_service.cc:1514] Processing DeleteTablet for tablet d3fddd068c67430da75b67741a9a6415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:05 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:47298
I20260929 02:13:05.436106 962 tablet_service.cc:1514] Processing DeleteTablet for tablet d3fddd068c67430da75b67741a9a6415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:05 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:43198
I20260929 02:13:05.448048 1128 tablet_replica.cc:330] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45: stopping tablet replica
I20260929 02:13:05.448398 1128 raft_consensus.cc:2229] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:13:05.448727 1128 raft_consensus.cc:2258] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:05.449980 1128 ts_tablet_manager.cc:1905] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:05.450742 1130 tablet_replica.cc:330] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1: stopping tablet replica
I20260929 02:13:05.450992 1130 raft_consensus.cc:2229] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:13:05.451275 1130 raft_consensus.cc:2258] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:05.452297 1130 ts_tablet_manager.cc:1905] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:05.453111 1129 tablet_replica.cc:330] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88: stopping tablet replica
I20260929 02:13:05.453338 1129 raft_consensus.cc:2229] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:13:05.453761 1129 raft_consensus.cc:2258] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:05.454870 1129 ts_tablet_manager.cc:1905] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:05.459754 1130 ts_tablet_manager.cc:1918] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:13:05.459950 1130 log.cc:1196] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/d3fddd068c67430da75b67741a9a6415
I20260929 02:13:05.460501 1130 ts_tablet_manager.cc:1939] T d3fddd068c67430da75b67741a9a6415 P 860d55bbf11e4d40a9149be66ef480a1: Deleting consensus metadata
I20260929 02:13:05.461660 532 catalog_manager.cc:4841] TS 860d55bbf11e4d40a9149be66ef480a1 (127.24.216.131:39709): tablet d3fddd068c67430da75b67741a9a6415 (table test-table [id=b8847fb9843f4c91955ae4f936e5b0c6]) successfully deleted
I20260929 02:13:05.462077 1128 ts_tablet_manager.cc:1918] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:13:05.462224 1128 log.cc:1196] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/d3fddd068c67430da75b67741a9a6415
I20260929 02:13:05.462714 1128 ts_tablet_manager.cc:1939] T d3fddd068c67430da75b67741a9a6415 P 4133caddbdd84b8dbc4ea2f3fff2de45: Deleting consensus metadata
I20260929 02:13:05.463768 534 catalog_manager.cc:4841] TS 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635): tablet d3fddd068c67430da75b67741a9a6415 (table test-table [id=b8847fb9843f4c91955ae4f936e5b0c6]) successfully deleted
I20260929 02:13:05.467619 1129 ts_tablet_manager.cc:1918] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:13:05.467768 1129 log.cc:1196] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/d3fddd068c67430da75b67741a9a6415
I20260929 02:13:05.468163 1129 ts_tablet_manager.cc:1939] T d3fddd068c67430da75b67741a9a6415 P ca6faaba97bf4b9691d4c6536df17c88: Deleting consensus metadata
I20260929 02:13:05.469053 532 catalog_manager.cc:4841] TS ca6faaba97bf4b9691d4c6536df17c88 (127.24.216.129:46435): tablet d3fddd068c67430da75b67741a9a6415 (table test-table [id=b8847fb9843f4c91955ae4f936e5b0c6]) successfully deleted
Sep 29 02:13:05 dist-test-slave-9k0h krb5kdc[503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:13:05 dist-test-slave-9k0h krb5kdc[503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:05 dist-test-slave-9k0h krb5kdc[503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:13:05.503269 1092 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:05.491823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.24.216.190:40503)
0929 02:13:05.492072 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:05.492081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:05.492461 (+ 380us) server_negotiation.cc:407] Connection header received
0929 02:13:05.492640 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:05.492644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:05.492706 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:05.492788 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:05.494452 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.495338 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.497176 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.497579 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.497674 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.502996 (+ 5322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43220: BlockingRecv error: recv got EOF from 127.0.0.1:43220 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 02:13:05 dist-test-slave-9k0h krb5kdc[503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM
I20260929 02:13:05.521078 1092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.506120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.24.216.190:40503)
0929 02:13:05.506310 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:05.506314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:05.506844 (+ 530us) server_negotiation.cc:407] Connection header received
0929 02:13:05.506990 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:05.506993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:05.507047 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:05.507114 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:05.508541 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.509485 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.511519 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.511969 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.512098 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.515946 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:05.515951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:05.515954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:05.515983 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:05.518499 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:05.519095 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:05.519105 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:05.519107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:05.519228 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:05.519664 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:05.519670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:05.519672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:05.519948 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:05.520208 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:05.520483 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 02:13:05.520719 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 02:13:05.524335 546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:13:05.524945 546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:13:05.531513 685 tablet_service.cc:1467] Processing CreateTablet for tablet 6173a9f3c1784828a1433e12a4b6b372 (DEFAULT_TABLE table=test-table [id=f2977c35de6a4c53b42c2d5b633c5deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:05.531934 685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6173a9f3c1784828a1433e12a4b6b372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:05.531883 962 tablet_service.cc:1467] Processing CreateTablet for tablet 6173a9f3c1784828a1433e12a4b6b372 (DEFAULT_TABLE table=test-table [id=f2977c35de6a4c53b42c2d5b633c5deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:05.532321 962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6173a9f3c1784828a1433e12a4b6b372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:05.538228 1058 tablet_bootstrap.cc:492] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88: Bootstrap starting.
I20260929 02:13:05.538698 1059 tablet_bootstrap.cc:492] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1: Bootstrap starting.
I20260929 02:13:05.538599 823 tablet_service.cc:1467] Processing CreateTablet for tablet 6173a9f3c1784828a1433e12a4b6b372 (DEFAULT_TABLE table=test-table [id=f2977c35de6a4c53b42c2d5b633c5deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:05.539026 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6173a9f3c1784828a1433e12a4b6b372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:05.539779 1058 tablet_bootstrap.cc:654] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:05.540364 1059 tablet_bootstrap.cc:654] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:05.547354 1057 tablet_bootstrap.cc:492] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45: Bootstrap starting.
I20260929 02:13:05.549392 1057 tablet_bootstrap.cc:654] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:05.550791 1059 tablet_bootstrap.cc:492] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1: No bootstrap required, opened a new log
I20260929 02:13:05.551017 1059 ts_tablet_manager.cc:1397] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.009s
I20260929 02:13:05.551898 1057 tablet_bootstrap.cc:492] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45: No bootstrap required, opened a new log
I20260929 02:13:05.552026 1057 ts_tablet_manager.cc:1397] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 02:13:05.551867 1059 raft_consensus.cc:348] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.552343 1059 raft_consensus.cc:374] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:05.552485 1059 raft_consensus.cc:729] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860d55bbf11e4d40a9149be66ef480a1, State: Initialized, Role: FOLLOWER
I20260929 02:13:05.552749 1057 raft_consensus.cc:348] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.552987 1057 raft_consensus.cc:374] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:05.553076 1057 raft_consensus.cc:729] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4133caddbdd84b8dbc4ea2f3fff2de45, State: Initialized, Role: FOLLOWER
I20260929 02:13:05.552824 1059 consensus_queue.cc:260] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.553279 1057 consensus_queue.cc:260] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.554999 1057 ts_tablet_manager.cc:1428] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 02:13:05.556384 1059 ts_tablet_manager.cc:1428] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 02:13:05.557245 1058 tablet_bootstrap.cc:492] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88: No bootstrap required, opened a new log
I20260929 02:13:05.557391 1058 ts_tablet_manager.cc:1397] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.012s
I20260929 02:13:05.558593 1058 raft_consensus.cc:348] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.558889 1058 raft_consensus.cc:374] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:05.558997 1058 raft_consensus.cc:729] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6faaba97bf4b9691d4c6536df17c88, State: Initialized, Role: FOLLOWER
I20260929 02:13:05.559229 1058 consensus_queue.cc:260] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.559878 1058 ts_tablet_manager.cc:1428] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:13:05.597430 1126 raft_consensus.cc:482] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:13:05.597654 1126 raft_consensus.cc:504] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.598580 1126 leader_election.cc:288] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca6faaba97bf4b9691d4c6536df17c88 (127.24.216.129:46435), 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635)
I20260929 02:13:05.599059 710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6173a9f3c1784828a1433e12a4b6b372" candidate_uuid: "860d55bbf11e4d40a9149be66ef480a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca6faaba97bf4b9691d4c6536df17c88" is_pre_election: true
I20260929 02:13:05.599105 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6173a9f3c1784828a1433e12a4b6b372" candidate_uuid: "860d55bbf11e4d40a9149be66ef480a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" is_pre_election: true
I20260929 02:13:05.599321 710 raft_consensus.cc:2454] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 860d55bbf11e4d40a9149be66ef480a1 in term 0.
I20260929 02:13:05.599395 843 raft_consensus.cc:2454] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 860d55bbf11e4d40a9149be66ef480a1 in term 0.
I20260929 02:13:05.599857 914 leader_election.cc:302] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 860d55bbf11e4d40a9149be66ef480a1, ca6faaba97bf4b9691d4c6536df17c88; no voters:
I20260929 02:13:05.600176 1126 raft_consensus.cc:2781] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:13:05.600319 1126 raft_consensus.cc:482] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:13:05.600437 1126 raft_consensus.cc:3037] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:05.603075 1126 raft_consensus.cc:504] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.603724 1126 leader_election.cc:288] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [CANDIDATE]: Term 1 election: Requested vote from peers ca6faaba97bf4b9691d4c6536df17c88 (127.24.216.129:46435), 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635)
I20260929 02:13:05.604128 710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6173a9f3c1784828a1433e12a4b6b372" candidate_uuid: "860d55bbf11e4d40a9149be66ef480a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca6faaba97bf4b9691d4c6536df17c88"
I20260929 02:13:05.604197 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6173a9f3c1784828a1433e12a4b6b372" candidate_uuid: "860d55bbf11e4d40a9149be66ef480a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45"
I20260929 02:13:05.604319 710 raft_consensus.cc:3037] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:05.604368 843 raft_consensus.cc:3037] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:05.607280 710 raft_consensus.cc:2454] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 860d55bbf11e4d40a9149be66ef480a1 in term 1.
I20260929 02:13:05.607280 843 raft_consensus.cc:2454] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 860d55bbf11e4d40a9149be66ef480a1 in term 1.
I20260929 02:13:05.607698 916 leader_election.cc:302] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4133caddbdd84b8dbc4ea2f3fff2de45, 860d55bbf11e4d40a9149be66ef480a1; no voters:
I20260929 02:13:05.608023 1126 raft_consensus.cc:2781] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:13:05.608227 1126 raft_consensus.cc:686] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 LEADER]: Becoming Leader. State: Replica: 860d55bbf11e4d40a9149be66ef480a1, State: Running, Role: LEADER
I20260929 02:13:05.608566 1126 consensus_queue.cc:237] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.611521 545 catalog_manager.cc:5495] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 860d55bbf11e4d40a9149be66ef480a1 (127.24.216.131). New cstate: current_term: 1 leader_uuid: "860d55bbf11e4d40a9149be66ef480a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } health_report { overall_health: HEALTHY } } }
I20260929 02:13:05.626765 1108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.621816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43212 (local address 127.24.216.131:39709)
0929 02:13:05.621962 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:05.621967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:05.622200 (+ 233us) server_negotiation.cc:407] Connection header received
0929 02:13:05.622462 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:05.622469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:05.622539 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:05.622632 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:13:05.622988 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.623768 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.625031 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.625460 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.625600 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.626027 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:13:05.626121 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:13:05.626258 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:05.626403 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 02:13:05.626473 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 02:13:05.629899 710 raft_consensus.cc:1261] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Refusing update from remote peer 860d55bbf11e4d40a9149be66ef480a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:13:05.629936 843 raft_consensus.cc:1261] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Refusing update from remote peer 860d55bbf11e4d40a9149be66ef480a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:13:05.630491 1126 consensus_queue.cc:1035] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:13:05.630720 1097 consensus_queue.cc:1035] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:13:05.651152 548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:13:05.651391 548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:13:05.654332 548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7: Sending DeleteTablet for 3 replicas of tablet 6173a9f3c1784828a1433e12a4b6b372
I20260929 02:13:05.654985 823 tablet_service.cc:1514] Processing DeleteTablet for tablet 6173a9f3c1784828a1433e12a4b6b372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:05 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:49348
I20260929 02:13:05.655045 685 tablet_service.cc:1514] Processing DeleteTablet for tablet 6173a9f3c1784828a1433e12a4b6b372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:05 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:47298
I20260929 02:13:05.655280 1128 tablet_replica.cc:330] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45: stopping tablet replica
I20260929 02:13:05.655463 1128 raft_consensus.cc:2229] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:13:05.655500 1129 tablet_replica.cc:330] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88: stopping tablet replica
I20260929 02:13:05.655642 1128 raft_consensus.cc:2258] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:05.655681 1129 raft_consensus.cc:2229] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:13:05.655884 1129 raft_consensus.cc:2258] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:05.656507 1128 ts_tablet_manager.cc:1905] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:05.656692 962 tablet_service.cc:1514] Processing DeleteTablet for tablet 6173a9f3c1784828a1433e12a4b6b372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:05 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:43198
I20260929 02:13:05.656963 1130 tablet_replica.cc:330] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1: stopping tablet replica
I20260929 02:13:05.657133 1130 raft_consensus.cc:2229] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:13:05.657608 1130 raft_consensus.cc:2258] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:05.659101 1130 ts_tablet_manager.cc:1905] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:05.663352 1128 ts_tablet_manager.cc:1918] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:13:05.663518 1128 log.cc:1196] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/6173a9f3c1784828a1433e12a4b6b372
I20260929 02:13:05.664888 1129 ts_tablet_manager.cc:1905] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:05.664812 548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:13:05.665495 548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:13:05.665935 1130 ts_tablet_manager.cc:1918] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:13:05.666086 1130 log.cc:1196] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/6173a9f3c1784828a1433e12a4b6b372
I20260929 02:13:05.666553 1130 ts_tablet_manager.cc:1939] T 6173a9f3c1784828a1433e12a4b6b372 P 860d55bbf11e4d40a9149be66ef480a1: Deleting consensus metadata
I20260929 02:13:05.666632 1128 ts_tablet_manager.cc:1939] T 6173a9f3c1784828a1433e12a4b6b372 P 4133caddbdd84b8dbc4ea2f3fff2de45: Deleting consensus metadata
I20260929 02:13:05.667670 534 catalog_manager.cc:4841] TS 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635): tablet 6173a9f3c1784828a1433e12a4b6b372 (table test-table [id=f2977c35de6a4c53b42c2d5b633c5deb]) successfully deleted
I20260929 02:13:05.667919 532 catalog_manager.cc:4841] TS 860d55bbf11e4d40a9149be66ef480a1 (127.24.216.131:39709): tablet 6173a9f3c1784828a1433e12a4b6b372 (table test-table [id=f2977c35de6a4c53b42c2d5b633c5deb]) successfully deleted
I20260929 02:13:05.672015 1129 ts_tablet_manager.cc:1918] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:13:05.672148 1129 log.cc:1196] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/6173a9f3c1784828a1433e12a4b6b372
I20260929 02:13:05.672502 1129 ts_tablet_manager.cc:1939] T 6173a9f3c1784828a1433e12a4b6b372 P ca6faaba97bf4b9691d4c6536df17c88: Deleting consensus metadata
I20260929 02:13:05.673702 685 tablet_service.cc:1467] Processing CreateTablet for tablet 7d23f73fa36d4613a6135b074cfa1931 (DEFAULT_TABLE table=test-table [id=d39f037638304a598d5bae521faa4979]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:05.673787 823 tablet_service.cc:1467] Processing CreateTablet for tablet 7d23f73fa36d4613a6135b074cfa1931 (DEFAULT_TABLE table=test-table [id=d39f037638304a598d5bae521faa4979]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:05.674129 685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d23f73fa36d4613a6135b074cfa1931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:05.674129 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d23f73fa36d4613a6135b074cfa1931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:05.674503 962 tablet_service.cc:1467] Processing CreateTablet for tablet 7d23f73fa36d4613a6135b074cfa1931 (DEFAULT_TABLE table=test-table [id=d39f037638304a598d5bae521faa4979]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:05.674875 962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d23f73fa36d4613a6135b074cfa1931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:05.680240 1057 tablet_bootstrap.cc:492] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45: Bootstrap starting.
I20260929 02:13:05.681509 1058 tablet_bootstrap.cc:492] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88: Bootstrap starting.
I20260929 02:13:05.681779 1057 tablet_bootstrap.cc:654] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:05.682020 1059 tablet_bootstrap.cc:492] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1: Bootstrap starting.
I20260929 02:13:05.683948 1058 tablet_bootstrap.cc:654] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:05.684106 1059 tablet_bootstrap.cc:654] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:05.686511 532 catalog_manager.cc:4841] TS ca6faaba97bf4b9691d4c6536df17c88 (127.24.216.129:46435): tablet 6173a9f3c1784828a1433e12a4b6b372 (table test-table [id=f2977c35de6a4c53b42c2d5b633c5deb]) successfully deleted
I20260929 02:13:05.691720 1059 tablet_bootstrap.cc:492] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1: No bootstrap required, opened a new log
I20260929 02:13:05.691952 1059 ts_tablet_manager.cc:1397] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 02:13:05.692742 1057 tablet_bootstrap.cc:492] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45: No bootstrap required, opened a new log
I20260929 02:13:05.692901 1057 ts_tablet_manager.cc:1397] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260929 02:13:05.692924 1059 raft_consensus.cc:348] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.693281 1059 raft_consensus.cc:374] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:05.693409 1059 raft_consensus.cc:729] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860d55bbf11e4d40a9149be66ef480a1, State: Initialized, Role: FOLLOWER
I20260929 02:13:05.693642 1057 raft_consensus.cc:348] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.693864 1057 raft_consensus.cc:374] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:05.693939 1057 raft_consensus.cc:729] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4133caddbdd84b8dbc4ea2f3fff2de45, State: Initialized, Role: FOLLOWER
I20260929 02:13:05.693734 1059 consensus_queue.cc:260] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.694226 1057 consensus_queue.cc:260] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.695278 1059 ts_tablet_manager.cc:1428] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:13:05.696411 1057 ts_tablet_manager.cc:1428] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:13:05.700687 1058 tablet_bootstrap.cc:492] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88: No bootstrap required, opened a new log
I20260929 02:13:05.700870 1058 ts_tablet_manager.cc:1397] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.007s
I20260929 02:13:05.701697 1058 raft_consensus.cc:348] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.701961 1058 raft_consensus.cc:374] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:05.702050 1058 raft_consensus.cc:729] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6faaba97bf4b9691d4c6536df17c88, State: Initialized, Role: FOLLOWER
I20260929 02:13:05.702385 1058 consensus_queue.cc:260] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.703470 1058 ts_tablet_manager.cc:1428] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 02:13:05.754024 1126 raft_consensus.cc:482] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:13:05.754263 1126 raft_consensus.cc:504] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.754963 1126 leader_election.cc:288] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca6faaba97bf4b9691d4c6536df17c88 (127.24.216.129:46435), 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635)
I20260929 02:13:05.755414 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d23f73fa36d4613a6135b074cfa1931" candidate_uuid: "860d55bbf11e4d40a9149be66ef480a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" is_pre_election: true
I20260929 02:13:05.755446 710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d23f73fa36d4613a6135b074cfa1931" candidate_uuid: "860d55bbf11e4d40a9149be66ef480a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca6faaba97bf4b9691d4c6536df17c88" is_pre_election: true
I20260929 02:13:05.755628 843 raft_consensus.cc:2454] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 860d55bbf11e4d40a9149be66ef480a1 in term 0.
I20260929 02:13:05.755676 710 raft_consensus.cc:2454] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 860d55bbf11e4d40a9149be66ef480a1 in term 0.
I20260929 02:13:05.756045 916 leader_election.cc:302] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4133caddbdd84b8dbc4ea2f3fff2de45, 860d55bbf11e4d40a9149be66ef480a1; no voters:
I20260929 02:13:05.756320 1126 raft_consensus.cc:2781] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:13:05.756467 1126 raft_consensus.cc:482] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:13:05.756593 1126 raft_consensus.cc:3037] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:05.758981 1126 raft_consensus.cc:504] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.759960 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d23f73fa36d4613a6135b074cfa1931" candidate_uuid: "860d55bbf11e4d40a9149be66ef480a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45"
I20260929 02:13:05.760048 710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d23f73fa36d4613a6135b074cfa1931" candidate_uuid: "860d55bbf11e4d40a9149be66ef480a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca6faaba97bf4b9691d4c6536df17c88"
I20260929 02:13:05.760149 843 raft_consensus.cc:3037] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:05.760239 710 raft_consensus.cc:3037] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:05.762240 843 raft_consensus.cc:2454] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 860d55bbf11e4d40a9149be66ef480a1 in term 1.
I20260929 02:13:05.762485 1126 leader_election.cc:288] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [CANDIDATE]: Term 1 election: Requested vote from peers ca6faaba97bf4b9691d4c6536df17c88 (127.24.216.129:46435), 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635)
I20260929 02:13:05.762681 916 leader_election.cc:302] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4133caddbdd84b8dbc4ea2f3fff2de45, 860d55bbf11e4d40a9149be66ef480a1; no voters:
I20260929 02:13:05.762931 1126 raft_consensus.cc:2781] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:13:05.763068 1126 raft_consensus.cc:686] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 LEADER]: Becoming Leader. State: Replica: 860d55bbf11e4d40a9149be66ef480a1, State: Running, Role: LEADER
I20260929 02:13:05.763193 710 raft_consensus.cc:2454] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 860d55bbf11e4d40a9149be66ef480a1 in term 1.
I20260929 02:13:05.763368 1126 consensus_queue.cc:237] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } }
I20260929 02:13:05.765913 548 catalog_manager.cc:5495] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 860d55bbf11e4d40a9149be66ef480a1 (127.24.216.131). New cstate: current_term: 1 leader_uuid: "860d55bbf11e4d40a9149be66ef480a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "860d55bbf11e4d40a9149be66ef480a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 39709 } health_report { overall_health: HEALTHY } } }
I20260929 02:13:05.802000 1108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.796772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:56321 (local address 127.24.216.131:39709)
0929 02:13:05.796951 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:05.796957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:05.796977 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:13:05.797114 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:05.797120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:05.797200 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:05.797322 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:13:05.797665 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.798779 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.800070 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.800764 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.800899 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.801259 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:13:05.801415 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:13:05.801505 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:05.801623 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 02:13:05.801693 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":57}
I20260929 02:13:05.802040 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.796705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:39709 (local address 127.24.216.131:56321)
0929 02:13:05.796886 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:05.796910 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:05.796992 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:05.797328 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:05.797332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:05.797340 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:13:05.797544 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:05.797553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.798911 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.798914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:05.799937 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:05.799947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.800938 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.800944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:05.801070 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.801105 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:13:05.801513 (+ 408us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:13:05.801520 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:13:05.801581 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 02:13:05.801672 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":20}
I20260929 02:13:05.809695 843 raft_consensus.cc:1261] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Refusing update from remote peer 860d55bbf11e4d40a9149be66ef480a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:13:05.809873 710 raft_consensus.cc:1261] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Refusing update from remote peer 860d55bbf11e4d40a9149be66ef480a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:13:05.810256 1103 consensus_queue.cc:1035] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4133caddbdd84b8dbc4ea2f3fff2de45" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 42635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:13:05.810489 1097 consensus_queue.cc:1035] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca6faaba97bf4b9691d4c6536df17c88" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:13:05.867451 548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:13:05.867704 548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:13:05.870538 548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af83698ec90144a08c569fb8763414b7: Sending DeleteTablet for 3 replicas of tablet 7d23f73fa36d4613a6135b074cfa1931
I20260929 02:13:05.871286 685 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d23f73fa36d4613a6135b074cfa1931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:05 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:47298
I20260929 02:13:05.871553 1129 tablet_replica.cc:330] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88: stopping tablet replica
I20260929 02:13:05.871658 963 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d23f73fa36d4613a6135b074cfa1931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:05 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:43198
I20260929 02:13:05.871747 1129 raft_consensus.cc:2229] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:13:05.871912 1130 tablet_replica.cc:330] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1: stopping tablet replica
I20260929 02:13:05.871948 1129 raft_consensus.cc:2258] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:05.872107 1130 raft_consensus.cc:2229] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:13:05.872411 1130 raft_consensus.cc:2258] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:05.872516 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 619
I20260929 02:13:05.872555 823 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d23f73fa36d4613a6135b074cfa1931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:05 UTC) from {username='oryx', principal='oryx/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:49348
I20260929 02:13:05.872828 1128 tablet_replica.cc:330] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45: stopping tablet replica
I20260929 02:13:05.872994 1128 raft_consensus.cc:2229] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:13:05.873171 1128 raft_consensus.cc:2258] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:05.873968 1128 ts_tablet_manager.cc:1905] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:05.875403 1130 ts_tablet_manager.cc:1905] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:05.880465 1128 ts_tablet_manager.cc:1918] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:13:05.880586 1128 log.cc:1196] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/7d23f73fa36d4613a6135b074cfa1931
I20260929 02:13:05.881013 1128 ts_tablet_manager.cc:1939] T 7d23f73fa36d4613a6135b074cfa1931 P 4133caddbdd84b8dbc4ea2f3fff2de45: Deleting consensus metadata
I20260929 02:13:05.881598 1130 ts_tablet_manager.cc:1918] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:13:05.881747 1130 log.cc:1196] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/7d23f73fa36d4613a6135b074cfa1931
I20260929 02:13:05.881991 534 catalog_manager.cc:4841] TS 4133caddbdd84b8dbc4ea2f3fff2de45 (127.24.216.130:42635): tablet 7d23f73fa36d4613a6135b074cfa1931 (table test-table [id=d39f037638304a598d5bae521faa4979]) successfully deleted
I20260929 02:13:05.882275 1130 ts_tablet_manager.cc:1939] T 7d23f73fa36d4613a6135b074cfa1931 P 860d55bbf11e4d40a9149be66ef480a1: Deleting consensus metadata
I20260929 02:13:05.883155 1129 ts_tablet_manager.cc:1905] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:05.883570 532 catalog_manager.cc:4841] TS 860d55bbf11e4d40a9149be66ef480a1 (127.24.216.131:39709): tablet 7d23f73fa36d4613a6135b074cfa1931 (table test-table [id=d39f037638304a598d5bae521faa4979]) successfully deleted
I20260929 02:13:05.889487 1129 ts_tablet_manager.cc:1918] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:13:05.889613 1129 log.cc:1196] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/7d23f73fa36d4613a6135b074cfa1931
I20260929 02:13:05.890036 1129 ts_tablet_manager.cc:1939] T 7d23f73fa36d4613a6135b074cfa1931 P ca6faaba97bf4b9691d4c6536df17c88: Deleting consensus metadata
I20260929 02:13:05.891005 532 catalog_manager.cc:4841] TS ca6faaba97bf4b9691d4c6536df17c88 (127.24.216.129:46435): tablet 7d23f73fa36d4613a6135b074cfa1931 (table test-table [id=d39f037638304a598d5bae521faa4979]) successfully deleted
Sep 29 02:13:05 dist-test-slave-9k0h krb5kdc[503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.216.129@KRBTEST.COM
I20260929 02:13:05.901823 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:05.873743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.24.216.129:46435)
0929 02:13:05.873921 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:05.873928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:05.888006 (+ 14078us) server_negotiation.cc:407] Connection header received
0929 02:13:05.888210 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:05.888232 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:05.888344 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:05.888467 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:05.889682 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.890576 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.891925 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:05.892305 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:05.892437 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:05.896450 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:05.896462 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:05.896477 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:05.896504 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:05.899248 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:05.900019 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:05.900025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:05.900027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:05.900137 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:05.900519 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:05.900522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:05.900524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:05.900827 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:05.901133 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:05.901342 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 02:13:05.901516 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:13:05.902451 750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:06.024106 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 619
I20260929 02:13:06.047281 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 759
Sep 29 02:13:06 dist-test-slave-9k0h krb5kdc[503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.216.130@KRBTEST.COM
I20260929 02:13:06.064193 1160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:06.048124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.24.216.130:42635)
0929 02:13:06.048595 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:06.048599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:06.048615 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:13:06.048669 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:06.048672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:06.048729 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:06.048815 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:06.050474 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:06.051174 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:06.052928 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:06.053330 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:06.053420 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:06.058558 (+ 5138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:06.058571 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:06.058587 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:06.058622 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:06.061477 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:06.062127 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:06.062134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:06.062136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:06.062317 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:06.062681 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:06.062688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:06.062691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:06.063056 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:06.063369 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:06.063620 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 02:13:06.063827 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":258,"threads_started":1}
I20260929 02:13:06.064818 889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:06.161433 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 759
I20260929 02:13:06.179338 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 898
Sep 29 02:13:06 dist-test-slave-9k0h krb5kdc[503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.216.131@KRBTEST.COM
I20260929 02:13:06.194221 1108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:06.180067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.24.216.131:39709)
0929 02:13:06.180266 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:06.180272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:06.180408 (+ 136us) server_negotiation.cc:407] Connection header received
0929 02:13:06.180610 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:06.180615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:06.180688 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:06.180817 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:06.181985 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:06.182885 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:06.184156 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:06.184657 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:06.184804 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:06.188966 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:06.188980 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:06.188996 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:06.189025 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:06.191914 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:06.192576 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:06.192583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:06.192585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:06.192718 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:06.193012 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:06.193015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:06.193017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:06.193328 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:06.193537 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:06.193731 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 02:13:06.193874 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:13:06.194898 1028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:06.343011 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:06.337890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:42635 (local address 127.24.216.131:37705)
0929 02:13:06.342684 (+ 4794us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:06.342811 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.130:42635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4674,"threads_started":1}
I20260929 02:13:06.343127 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 898
I20260929 02:13:06.368922 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 516
I20260929 02:13:06.370234 588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:06.482957 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 516
2026-09-29T02:13:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5580 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:13:06 dist-test-slave-9k0h krb5kdc[1168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:13:06 dist-test-slave-9k0h krb5kdc[1168](info): set up 2 sockets
Sep 29 02:13:06 dist-test-slave-9k0h krb5kdc[1168](info): commencing operation
krb5kdc: starting...
W20260929 02:13:07.563060 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:13:07 dist-test-slave-9k0h krb5kdc[1168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:13:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:13:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:13:07.776000 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:44527
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:44527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:13:08.091131 1181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:08.091398 1181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:08.091485 1181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:08.098353 1181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:13:08.098443 1181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:08.098495 1181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:08.098548 1181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:13:08.098594 1181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:13:08.107725 1181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:44527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:44527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:08.108855 1181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:08.110093 1181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:08.121282 1186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:08.121335 1187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:08.121356 1189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:08.123458 1181 server_base.cc:1034] running on GCE node
I20260929 02:13:08.124181 1181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:08.125504 1181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:08.126730 1181 hybrid_clock.cc:648] HybridClock initialized: now 1790647988126669 us; error 51 us; skew 500 ppm
I20260929 02:13:08.127071 1181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T02:13:08Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1709 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:13:08 dist-test-slave-9k0h krb5kdc[1195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:13:08 dist-test-slave-9k0h krb5kdc[1195](info): set up 2 sockets
Sep 29 02:13:08 dist-test-slave-9k0h krb5kdc[1195](info): commencing operation
krb5kdc: starting...
W20260929 02:13:09.295441 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:13:09 dist-test-slave-9k0h krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:13:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:13:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:13:09.513447 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:36003
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:36003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:13:09.834525 1208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:09.834802 1208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:09.834972 1208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:09.841895 1208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:13:09.841995 1208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:09.842095 1208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:09.842211 1208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:13:09.842303 1208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:13:09.851925 1208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:36003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:36003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:09.853178 1208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:09.854527 1208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:09.866066 1213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:09.866020 1214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:09.866291 1216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:09.867118 1208 server_base.cc:1034] running on GCE node
I20260929 02:13:09.867828 1208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:09.869050 1208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:09.870261 1208 hybrid_clock.cc:648] HybridClock initialized: now 1790647989870183 us; error 51 us; skew 500 ppm
I20260929 02:13:09.870553 1208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:09 dist-test-slave-9k0h krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647989, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:09.873520 1208 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:13:09.874943 1208 webserver.cc:460] Webserver started at http://127.24.216.190:43711/ using document root <none> and password file <none>
I20260929 02:13:09.875584 1208 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:09.875708 1208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:09.876021 1208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:09.878899 1208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "29b63b7b80204b9895a85276b138de2a"
format_stamp: "Formatted at 2026-09-29 02:13:09 on dist-test-slave-9k0h"
server_key: "7ca462f4eb2fc5186811b9e73695cc8b"
server_key_iv: "3c09d924fe6e9c395c9c5f8d3669d5f4"
server_key_version: "encryptionkey@0"
I20260929 02:13:09.879703 1208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "29b63b7b80204b9895a85276b138de2a"
format_stamp: "Formatted at 2026-09-29 02:13:09 on dist-test-slave-9k0h"
server_key: "7ca462f4eb2fc5186811b9e73695cc8b"
server_key_iv: "3c09d924fe6e9c395c9c5f8d3669d5f4"
server_key_version: "encryptionkey@0"
I20260929 02:13:09.884495 1208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:13:09.887835 1223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:09.889101 1208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:13:09.889259 1208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "29b63b7b80204b9895a85276b138de2a"
format_stamp: "Formatted at 2026-09-29 02:13:09 on dist-test-slave-9k0h"
server_key: "7ca462f4eb2fc5186811b9e73695cc8b"
server_key_iv: "3c09d924fe6e9c395c9c5f8d3669d5f4"
server_key_version: "encryptionkey@0"
I20260929 02:13:09.889443 1208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:09.904582 1208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:09.908504 1208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:09.908811 1208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:09.927027 1208 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:36003
I20260929 02:13:09.927062 1274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:36003 every 8 connection(s)
I20260929 02:13:09.928363 1208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:13:09.931975 1275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:09.936354 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 1208
I20260929 02:13:09.936553 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:13:09.937239 25442 external_mini_cluster.cc:1428] Setting key 568e48dec105ef32423b93cd1cbfe6a1
I20260929 02:13:09.942323 1275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a: Bootstrap starting.
I20260929 02:13:09.945787 1275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:09.946969 1275 log.cc:824] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:13:09.950330 1275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a: No bootstrap required, opened a new log
Sep 29 02:13:09 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:13:09.957892 1275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b63b7b80204b9895a85276b138de2a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36003 } }
I20260929 02:13:09.958324 1275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:09.958452 1275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b63b7b80204b9895a85276b138de2a, State: Initialized, Role: FOLLOWER
I20260929 02:13:09.959045 1275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b63b7b80204b9895a85276b138de2a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36003 } }
I20260929 02:13:09.959296 1275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:13:09.959473 1275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:13:09.959658 1275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:09.960177 1278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:09.939125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37128 (local address 127.24.216.190:36003)
0929 02:13:09.939952 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:09.939963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:09.940010 (+ 47us) server_negotiation.cc:407] Connection header received
0929 02:13:09.941635 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:09.941657 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:09.942014 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:09.942377 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:09.944125 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:09.945331 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:09.947136 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:09.948172 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:09.948337 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:09.952875 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:09.952886 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:09.952899 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:09.952930 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:09.956068 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:09.956752 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:09.956758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:09.956767 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:09.956921 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:09.957294 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:09.957298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:09.957300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:09.957780 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:09.958095 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:09.958603 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 02:13:09.959007 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":395,"threads_started":1}
I20260929 02:13:09.962014 1275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b63b7b80204b9895a85276b138de2a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36003 } }
I20260929 02:13:09.962541 1275 leader_election.cc:302] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29b63b7b80204b9895a85276b138de2a; no voters:
I20260929 02:13:09.963248 1275 leader_election.cc:288] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:13:09.963335 1280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:13:09.964001 1280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 1 LEADER]: Becoming Leader. State: Replica: 29b63b7b80204b9895a85276b138de2a, State: Running, Role: LEADER
I20260929 02:13:09.964560 1280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b63b7b80204b9895a85276b138de2a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36003 } }
I20260929 02:13:09.965402 1275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:13:09.967231 1281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29b63b7b80204b9895a85276b138de2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b63b7b80204b9895a85276b138de2a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36003 } } }
I20260929 02:13:09.967746 1281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [sys.catalog]: This master's current role is: LEADER
I20260929 02:13:09.967608 1282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b63b7b80204b9895a85276b138de2a. Latest consensus state: current_term: 1 leader_uuid: "29b63b7b80204b9895a85276b138de2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b63b7b80204b9895a85276b138de2a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36003 } } }
I20260929 02:13:09.967866 1282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [sys.catalog]: This master's current role is: LEADER
I20260929 02:13:09.968706 1287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:13:09.973788 1287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:13:09.982832 1287 catalog_manager.cc:1347] Generated new cluster ID: bd5a4dc198d94cef9103d50208c3dc1f
I20260929 02:13:09.982946 1287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:13:10.003750 1287 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:13:10.005060 1287 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:13:10.012868 1287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a: Generated new TSK 0
I20260929 02:13:10.013748 1287 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:13:10.077340 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36003
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:13:10.405665 1301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:10.405929 1301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:10.406092 1301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:10.413383 1301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:10.413482 1301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:10.413671 1301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:13:10.423400 1301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:10.424650 1301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:10.425987 1301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:10.438932 1309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:10.439045 1307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:10.439033 1306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:10.445202 1301 server_base.cc:1034] running on GCE node
I20260929 02:13:10.445811 1301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:10.446781 1301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:10.447980 1301 hybrid_clock.cc:648] HybridClock initialized: now 1790647990447921 us; error 54 us; skew 500 ppm
I20260929 02:13:10.448294 1301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:10 dist-test-slave-9k0h krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647990, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:10.456012 1301 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:13:10.457525 1301 webserver.cc:460] Webserver started at http://127.24.216.129:45009/ using document root <none> and password file <none>
I20260929 02:13:10.458323 1301 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:10.458467 1301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:10.458776 1301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:10.461877 1301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "c0fe033048914be5bfda30daeeba215c"
format_stamp: "Formatted at 2026-09-29 02:13:10 on dist-test-slave-9k0h"
server_key: "5a53517836d31a5feeab6a2d41af383e"
server_key_iv: "fcc976f4ace4fee7341ff8a43f69be14"
server_key_version: "encryptionkey@0"
I20260929 02:13:10.462729 1301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "c0fe033048914be5bfda30daeeba215c"
format_stamp: "Formatted at 2026-09-29 02:13:10 on dist-test-slave-9k0h"
server_key: "5a53517836d31a5feeab6a2d41af383e"
server_key_iv: "fcc976f4ace4fee7341ff8a43f69be14"
server_key_version: "encryptionkey@0"
I20260929 02:13:10.468080 1301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:13:10.471832 1316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:10.473098 1301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:13:10.473266 1301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "c0fe033048914be5bfda30daeeba215c"
format_stamp: "Formatted at 2026-09-29 02:13:10 on dist-test-slave-9k0h"
server_key: "5a53517836d31a5feeab6a2d41af383e"
server_key_iv: "fcc976f4ace4fee7341ff8a43f69be14"
server_key_version: "encryptionkey@0"
I20260929 02:13:10.473472 1301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:10.495720 1301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:10.499634 1301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:10.499908 1301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:10.500813 1301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:10.502056 1301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:10.502136 1301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:10.502338 1301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:10.502398 1301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:10.535933 1301 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:42551
I20260929 02:13:10.535979 1428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:42551 every 8 connection(s)
I20260929 02:13:10.537282 1301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:13:10.539781 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 1301
I20260929 02:13:10.540004 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:13:10.540781 25442 external_mini_cluster.cc:1428] Setting key 70797b521cf93075c48140076b851214
Sep 29 02:13:10 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647990, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:13:10.558810 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:10.539810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:53833 (local address 127.24.216.190:36003)
0929 02:13:10.540312 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:10.540319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:10.541373 (+ 1054us) server_negotiation.cc:407] Connection header received
0929 02:13:10.543579 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:10.543587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:10.543672 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:10.543801 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:10.545192 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:10.546292 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:10.547732 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:10.548467 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:10.548614 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:10.551985 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:10.551994 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:10.552002 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:10.552043 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:10.555609 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:10.556361 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:10.556370 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:10.556376 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:10.556514 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:10.556897 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:10.556905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:10.556911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:10.557264 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:10.557432 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:10.558053 (+ 621us) server_negotiation.cc:299] Negotiation successful
0929 02:13:10.558330 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":234,"threads_started":1}
I20260929 02:13:10.559423 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:10.540397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:53833)
0929 02:13:10.541188 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:10.541244 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:10.543309 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:10.543990 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:10.543997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:10.544483 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:10.545017 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:10.545034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:10.546439 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:10.546443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:10.547594 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:10.547604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:10.548665 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:10.548669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:10.548856 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:10.549541 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:10.549565 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:10.551786 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:10.555837 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:10.555843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:10.555853 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:10.556239 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:10.556624 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:10.556626 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:10.556629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:10.556787 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:10.557387 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:10.557392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:10.557693 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 02:13:10.557907 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 02:13:10.558340 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":283,"threads_started":1}
I20260929 02:13:10.560940 1429 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36003
I20260929 02:13:10.561246 1429 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:10.561920 1429 heartbeater.cc:511] Master 127.24.216.190:36003 requested a full tablet report, sending...
I20260929 02:13:10.563961 1240 ts_manager.cc:194] Registered new tserver with Master: c0fe033048914be5bfda30daeeba215c (127.24.216.129:42551)
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
I20260929 02:13:10.565498 1240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:53833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:13:10.593004 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36003
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:13:10.912009 1435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:10.912257 1435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:10.912369 1435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:10.919567 1435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:10.919673 1435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:10.919848 1435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:13:10.929247 1435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:10.930501 1435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:10.931823 1435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:10.945154 1443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:10.945199 1441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:10.945118 1440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:10.946823 1435 server_base.cc:1034] running on GCE node
I20260929 02:13:10.947425 1435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:10.948282 1435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:10.949472 1435 hybrid_clock.cc:648] HybridClock initialized: now 1790647990949430 us; error 32 us; skew 500 ppm
I20260929 02:13:10.949775 1435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:10 dist-test-slave-9k0h krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647990, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:10.952771 1435 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:13:10.954053 1435 webserver.cc:460] Webserver started at http://127.24.216.130:46203/ using document root <none> and password file <none>
I20260929 02:13:10.954756 1435 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:10.954877 1435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:10.955181 1435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:10.957890 1435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "7f35f9b5f7e8468498d7c5908d97a144"
format_stamp: "Formatted at 2026-09-29 02:13:10 on dist-test-slave-9k0h"
server_key: "fd61bb09140d811e5a166e22dff4d3a1"
server_key_iv: "bee7a5c14221f640805e9ab73d235cb2"
server_key_version: "encryptionkey@0"
I20260929 02:13:10.958673 1435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "7f35f9b5f7e8468498d7c5908d97a144"
format_stamp: "Formatted at 2026-09-29 02:13:10 on dist-test-slave-9k0h"
server_key: "fd61bb09140d811e5a166e22dff4d3a1"
server_key_iv: "bee7a5c14221f640805e9ab73d235cb2"
server_key_version: "encryptionkey@0"
I20260929 02:13:10.963549 1435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 02:13:10.966850 1450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:10.968046 1435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:13:10.968207 1435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "7f35f9b5f7e8468498d7c5908d97a144"
format_stamp: "Formatted at 2026-09-29 02:13:10 on dist-test-slave-9k0h"
server_key: "fd61bb09140d811e5a166e22dff4d3a1"
server_key_iv: "bee7a5c14221f640805e9ab73d235cb2"
server_key_version: "encryptionkey@0"
I20260929 02:13:10.968391 1435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:10.990419 1435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:10.994760 1435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:10.995034 1435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:10.995937 1435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:10.997329 1435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:10.997430 1435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:10.997576 1435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:10.997648 1435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:11.030472 1435 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:44365
I20260929 02:13:11.030555 1562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:44365 every 8 connection(s)
I20260929 02:13:11.031775 1435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:13:11.034097 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 1435
I20260929 02:13:11.034292 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:13:11.034830 25442 external_mini_cluster.cc:1428] Setting key d74b91233e27ab34703c4408f5def98b
Sep 29 02:13:11 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647990, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:13:11.053333 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:11.034495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:39867 (local address 127.24.216.190:36003)
0929 02:13:11.034704 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:11.034711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:11.036002 (+ 1291us) server_negotiation.cc:407] Connection header received
0929 02:13:11.037830 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:11.037838 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:11.037926 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:11.038034 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:11.039652 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:11.040369 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:11.041883 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:11.042372 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:11.042508 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:11.046021 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:11.046029 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:11.046037 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:11.046082 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:11.049925 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:11.050820 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:11.050828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:11.050834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:11.050992 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:11.051397 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:11.051404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:11.051410 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:11.051781 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:11.051926 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:11.052594 (+ 668us) server_negotiation.cc:299] Negotiation successful
0929 02:13:11.052867 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:13:11.053918 1565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:11.035041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:39867)
0929 02:13:11.035833 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:11.035879 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:11.037559 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:11.038325 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:11.038334 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:11.038868 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:11.039466 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:11.039483 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:11.040526 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:11.040531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:11.041743 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:11.041754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:11.042587 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:11.042594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:11.042824 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:11.043503 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:11.043524 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:11.045813 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:11.050213 (+ 4400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:11.050225 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:11.050248 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:11.050693 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:11.051104 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:11.051107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:11.051109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:11.051273 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:11.051893 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:11.051898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:11.052209 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 02:13:11.052439 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 02:13:11.052807 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":329,"threads_started":1}
I20260929 02:13:11.055397 1563 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36003
I20260929 02:13:11.055707 1563 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:11.056389 1563 heartbeater.cc:511] Master 127.24.216.190:36003 requested a full tablet report, sending...
I20260929 02:13:11.057806 1240 ts_manager.cc:194] Registered new tserver with Master: 7f35f9b5f7e8468498d7c5908d97a144 (127.24.216.130:44365)
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
I20260929 02:13:11.058646 1240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:39867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:13:11.087708 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36003
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:13:11.431553 1568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:11.431958 1568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:11.432124 1568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:11.443282 1568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:11.443405 1568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:11.443603 1568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:13:11.455016 1568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:11.456295 1568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:11.457545 1568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:11.470887 1573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:11.470939 1574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:11.471040 1576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:11.471905 1568 server_base.cc:1034] running on GCE node
I20260929 02:13:11.472535 1568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:11.473387 1568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:11.474587 1568 hybrid_clock.cc:648] HybridClock initialized: now 1790647991474547 us; error 34 us; skew 500 ppm
I20260929 02:13:11.474924 1568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:11 dist-test-slave-9k0h krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647991, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:11.477983 1568 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:13:11.479471 1568 webserver.cc:460] Webserver started at http://127.24.216.131:36933/ using document root <none> and password file <none>
I20260929 02:13:11.480131 1568 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:11.480273 1568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:11.480578 1568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:11.483414 1568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "e5b9be886df44e0fa32086e67e5e3630"
format_stamp: "Formatted at 2026-09-29 02:13:11 on dist-test-slave-9k0h"
server_key: "eb3dcfe003ffeb75691b875e4cebbe79"
server_key_iv: "28240e99ddf8600e1367ceabe66f1db1"
server_key_version: "encryptionkey@0"
I20260929 02:13:11.484151 1568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "e5b9be886df44e0fa32086e67e5e3630"
format_stamp: "Formatted at 2026-09-29 02:13:11 on dist-test-slave-9k0h"
server_key: "eb3dcfe003ffeb75691b875e4cebbe79"
server_key_iv: "28240e99ddf8600e1367ceabe66f1db1"
server_key_version: "encryptionkey@0"
I20260929 02:13:11.488981 1568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 02:13:11.492201 1583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:11.493535 1568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:13:11.493695 1568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "e5b9be886df44e0fa32086e67e5e3630"
format_stamp: "Formatted at 2026-09-29 02:13:11 on dist-test-slave-9k0h"
server_key: "eb3dcfe003ffeb75691b875e4cebbe79"
server_key_iv: "28240e99ddf8600e1367ceabe66f1db1"
server_key_version: "encryptionkey@0"
I20260929 02:13:11.493939 1568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:11.511620 1568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:11.515344 1568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:11.515630 1568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:11.516517 1568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:11.517784 1568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:11.517889 1568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:11.518043 1568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:11.518118 1568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:11.553634 1568 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:35541
I20260929 02:13:11.553681 1695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:35541 every 8 connection(s)
I20260929 02:13:11.555048 1568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:13:11.559988 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 1568
I20260929 02:13:11.560143 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:13:11.560599 25442 external_mini_cluster.cc:1428] Setting key c117e5ca29d5c15f4331ad7466c19453
Sep 29 02:13:11 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647991, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:13:11.568754 1429 heartbeater.cc:503] Master 127.24.216.190:36003 was elected leader, sending a full tablet report...
I20260929 02:13:11.575599 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:11.557409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:48993 (local address 127.24.216.190:36003)
0929 02:13:11.557836 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:11.557842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:11.558869 (+ 1027us) server_negotiation.cc:407] Connection header received
0929 02:13:11.560709 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:11.560715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:11.560809 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:11.560926 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:11.563213 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:11.564023 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:11.565609 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:11.566057 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:11.566254 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:11.569580 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:11.569588 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:11.569600 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:11.569647 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:11.572479 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:11.573167 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:11.573171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:11.573174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:11.573290 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:11.573655 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:11.573659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:11.573662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:11.573981 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:11.574137 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:11.575061 (+ 924us) server_negotiation.cc:299] Negotiation successful
0929 02:13:11.575230 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":209,"threads_started":1}
I20260929 02:13:11.576534 1699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:11.558001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:48993)
0929 02:13:11.558712 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:11.558749 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:11.560442 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:11.561159 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:11.561167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:11.561783 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:11.562486 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:11.562506 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:11.564168 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:11.564173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:11.565470 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:11.565481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:11.566336 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:11.566343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:11.566564 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:11.567201 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:11.567227 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:11.569394 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:11.572592 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:11.572598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:11.572609 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:11.573070 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:11.573383 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:11.573386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:11.573388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:11.573542 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:11.574110 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:11.574118 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:11.574504 (+ 386us) client_negotiation.cc:769] Sending connection context
0929 02:13:11.574826 (+ 322us) client_negotiation.cc:240] Negotiation successful
0929 02:13:11.575255 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":306,"threads_started":1}
I20260929 02:13:11.577840 1696 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36003
I20260929 02:13:11.578078 1696 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:11.578797 1696 heartbeater.cc:511] Master 127.24.216.190:36003 requested a full tablet report, sending...
I20260929 02:13:11.579990 1240 ts_manager.cc:194] Registered new tserver with Master: e5b9be886df44e0fa32086e67e5e3630 (127.24.216.131:35541)
I20260929 02:13:11.580710 1240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:48993
I20260929 02:13:11.587589 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.129.keytab.
Entry for principal oryx/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.129.keytab.
WARNING: no policy specified for oryx/127.24.216.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.130.keytab.
Entry for principal oryx/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.130.keytab.
WARNING: no policy specified for oryx/127.24.216.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.131.keytab.
Entry for principal oryx/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.131.keytab.
I20260929 02:13:11.724746 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 1301
Sep 29 02:13:11 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:13:11.740581 1707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:11.725967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.24.216.129:42551)
0929 02:13:11.726626 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:11.726633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:11.726659 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:13:11.726719 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:11.726725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:11.726978 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:11.727105 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:11.728320 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:11.729271 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:11.730589 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:11.731145 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:11.731241 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:11.735457 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:11.735467 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:11.735482 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:11.735511 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:11.738288 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:11.738810 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:11.738817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:11.738820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:11.738931 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:11.739245 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:11.739248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:11.739250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:11.739580 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:11.739795 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:11.739995 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 02:13:11.740174 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":364,"threads_started":1}
I20260929 02:13:11.741844 1424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:11.822417 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 1301
I20260929 02:13:11.838871 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 1435
Sep 29 02:13:11 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:13:11.853245 1709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:11.839754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38642 (local address 127.24.216.130:44365)
0929 02:13:11.840375 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:11.840382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:11.840407 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:13:11.840470 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:11.840476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:11.840693 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:11.840831 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:11.841941 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:11.842967 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:11.844107 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:11.844644 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:11.844733 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:11.848386 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:11.848397 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:11.848411 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:11.848446 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:11.851013 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:11.851608 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:11.851615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:11.851617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:11.851735 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:11.852020 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:11.852023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:11.852025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:11.852332 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:11.852545 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:11.852792 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 02:13:11.852979 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":333,"threads_started":1}
I20260929 02:13:11.854511 1558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:11.933071 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 1435
I20260929 02:13:11.949762 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 1568
Sep 29 02:13:11 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:13:11.964908 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:11.950654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54908 (local address 127.24.216.131:35541)
0929 02:13:11.951272 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:11.951279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:11.951308 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:13:11.951378 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:11.951384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:11.951587 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:11.951712 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:11.953044 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:11.953995 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:11.955319 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:11.955858 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:11.955951 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:11.959955 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:11.959965 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:11.959978 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:11.960008 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:11.962697 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:11.963269 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:11.963275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:11.963277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:11.963380 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:11.963665 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:11.963668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:11.963670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:11.963972 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:11.964197 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:11.964364 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 02:13:11.964537 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":311,"threads_started":1}
I20260929 02:13:11.966207 1691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:12.042889 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 1568
I20260929 02:13:12.058130 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 1208
I20260929 02:13:12.059303 1270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:12.159210 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 1208
I20260929 02:13:12.179782 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:36003
--webserver_interface=127.24.216.190
--webserver_port=43711
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:36003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:13:12.499485 1714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:12.499739 1714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:12.499823 1714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:12.506580 1714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:13:12.506683 1714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:12.506726 1714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:12.506779 1714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:13:12.506821 1714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:13:12.515650 1714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:36003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:36003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=43711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:12.516783 1714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:12.517961 1714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:12.529254 1719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:12.529165 1722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:12.529320 1720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:12.530333 1714 server_base.cc:1034] running on GCE node
I20260929 02:13:12.531040 1714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:12.532310 1714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:12.533526 1714 hybrid_clock.cc:648] HybridClock initialized: now 1790647992533482 us; error 53 us; skew 500 ppm
I20260929 02:13:12.533869 1714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:12 dist-test-slave-9k0h krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647992, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:12.537700 1714 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:13:12.539186 1714 webserver.cc:460] Webserver started at http://127.24.216.190:43711/ using document root <none> and password file <none>
I20260929 02:13:12.539929 1714 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:12.540086 1714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:12.545075 1714 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 02:13:12.547648 1729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:12.548825 1714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 02:13:12.549007 1714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "29b63b7b80204b9895a85276b138de2a"
format_stamp: "Formatted at 2026-09-29 02:13:09 on dist-test-slave-9k0h"
server_key: "7ca462f4eb2fc5186811b9e73695cc8b"
server_key_iv: "3c09d924fe6e9c395c9c5f8d3669d5f4"
server_key_version: "encryptionkey@0"
I20260929 02:13:12.549963 1714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:12.568898 1714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:12.572634 1714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:12.572881 1714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:12.591146 1714 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:36003
I20260929 02:13:12.591178 1780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:36003 every 8 connection(s)
I20260929 02:13:12.592592 1714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:13:12.599002 1781 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:13:12.600904 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 1714
I20260929 02:13:12.601670 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:42551
--local_ip_for_outbound_sockets=127.24.216.129
--tserver_master_addrs=127.24.216.190:36003
--webserver_port=45009
--webserver_interface=127.24.216.129
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.216.129
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:13:12.602284 1781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a: Bootstrap starting.
I20260929 02:13:12.624763 1781 log.cc:824] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:13:12.635337 1781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:13:12.635975 1781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a: Bootstrap complete.
I20260929 02:13:12.648723 1781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b63b7b80204b9895a85276b138de2a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36003 } }
I20260929 02:13:12.649811 1781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b63b7b80204b9895a85276b138de2a, State: Initialized, Role: FOLLOWER
I20260929 02:13:12.650588 1781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b63b7b80204b9895a85276b138de2a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36003 } }
I20260929 02:13:12.650853 1781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:13:12.650969 1781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:13:12.651136 1781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:13:12.656230 1781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b63b7b80204b9895a85276b138de2a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36003 } }
I20260929 02:13:12.656828 1781 leader_election.cc:302] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29b63b7b80204b9895a85276b138de2a; no voters:
I20260929 02:13:12.657408 1781 leader_election.cc:288] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:13:12.657559 1785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:13:12.658957 1785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [term 2 LEADER]: Becoming Leader. State: Replica: 29b63b7b80204b9895a85276b138de2a, State: Running, Role: LEADER
I20260929 02:13:12.659478 1781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:13:12.659509 1785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b63b7b80204b9895a85276b138de2a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36003 } }
I20260929 02:13:12.661954 1787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b63b7b80204b9895a85276b138de2a. Latest consensus state: current_term: 2 leader_uuid: "29b63b7b80204b9895a85276b138de2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b63b7b80204b9895a85276b138de2a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36003 } } }
I20260929 02:13:12.662344 1787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [sys.catalog]: This master's current role is: LEADER
I20260929 02:13:12.662839 1786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "29b63b7b80204b9895a85276b138de2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b63b7b80204b9895a85276b138de2a" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36003 } } }
I20260929 02:13:12.663153 1786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a [sys.catalog]: This master's current role is: LEADER
I20260929 02:13:12.666585 1793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:13:12.672290 1793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:13:12.673985 1793 catalog_manager.cc:1259] Loaded cluster ID: bd5a4dc198d94cef9103d50208c3dc1f
I20260929 02:13:12.674173 1793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:13:12.679221 1793 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:13:12.681236 1793 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 29b63b7b80204b9895a85276b138de2a: Loaded TSK: 0
I20260929 02:13:12.683218 1793 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:13:12.961176 1783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:12.961434 1783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:12.961568 1783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:12.968780 1783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:12.968875 1783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:12.969036 1783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:13:12.978504 1783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.129.keytab
--principal=oryx/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:42551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=45009
--tserver_master_addrs=127.24.216.190:36003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:12.979781 1783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:12.981047 1783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:12.993804 1808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:12.993780 1807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:12.994675 1810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:12.995085 1783 server_base.cc:1034] running on GCE node
I20260929 02:13:12.995640 1783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:12.996640 1783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:12.997856 1783 hybrid_clock.cc:648] HybridClock initialized: now 1790647992997805 us; error 47 us; skew 500 ppm
I20260929 02:13:12.998179 1783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:12 dist-test-slave-9k0h krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647992, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:13.001312 1783 init.cc:377] Logged in from keytab as oryx/127.24.216.129@KRBTEST.COM (short username oryx)
I20260929 02:13:13.002978 1783 webserver.cc:460] Webserver started at http://127.24.216.129:45009/ using document root <none> and password file <none>
I20260929 02:13:13.003604 1783 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:13.003765 1783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:13.008980 1783 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 02:13:13.011708 1817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:13.013053 1783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:13:13.013203 1783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "c0fe033048914be5bfda30daeeba215c"
format_stamp: "Formatted at 2026-09-29 02:13:10 on dist-test-slave-9k0h"
server_key: "5a53517836d31a5feeab6a2d41af383e"
server_key_iv: "fcc976f4ace4fee7341ff8a43f69be14"
server_key_version: "encryptionkey@0"
I20260929 02:13:13.014068 1783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:13.030684 1783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:13.034897 1783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:13.035158 1783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:13.036083 1783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:13.037371 1783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:13.037461 1783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:13.037611 1783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:13.037709 1783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:13.069517 1783 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:42551
I20260929 02:13:13.069605 1929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:42551 every 8 connection(s)
I20260929 02:13:13.071012 1783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:13:13.078087 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 1783
I20260929 02:13:13.078699 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:44365
--local_ip_for_outbound_sockets=127.24.216.130
--tserver_master_addrs=127.24.216.190:36003
--webserver_port=46203
--webserver_interface=127.24.216.130
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.216.130
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:13:13.088929 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:13.073732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:60715 (local address 127.24.216.190:36003)
0929 02:13:13.074426 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:13.074438 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:13.074844 (+ 406us) server_negotiation.cc:407] Connection header received
0929 02:13:13.076845 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:13.076858 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:13.077165 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:13.077450 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:13.078890 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.080152 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.081649 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.082361 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.082554 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.088085 (+ 5531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:60715: BlockingRecv error: recv got EOF from 127.24.216.129:60715 (error 108)
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":261,"threads_started":1}
I20260929 02:13:13.089013 1933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:13.073864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:60715)
0929 02:13:13.074668 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:13.074722 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:13.076365 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:13.077612 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:13.077620 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:13.078093 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:13.078703 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.078720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.080317 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.080323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.081505 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.081516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.082581 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.082587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.082866 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.083510 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:13.083530 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:13.087801 (+ 4271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":379,"threads_started":1}
W20260929 02:13:13.089808 1930 heartbeater.cc:644] Failed to heartbeat to 127.24.216.190:36003 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.216.190:36003: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:13.097689 1933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:13.090455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:53201)
0929 02:13:13.090620 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:13.090635 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:13.090731 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:13.091073 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:13.091077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:13.091410 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:13.091542 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.091550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.092535 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.092541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.093402 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.093412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.094116 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.094121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.094305 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.094879 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:13.094895 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:13.097429 (+ 2534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 02:13:13.097810 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:13.090654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:53201 (local address 127.24.216.190:36003)
0929 02:13:13.090843 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:13.090848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:13.090862 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:13:13.090907 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:13.090911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:13.090958 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:13.091054 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:13.091674 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.092399 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.093566 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.093976 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.094093 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.097597 (+ 3504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:53201: BlockingRecv error: recv got EOF from 127.24.216.129:53201 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:13.107162 1933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:13.098430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:47293)
0929 02:13:13.098612 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:13.098632 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:13.098735 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:13.099035 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:13.099041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:13.099483 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:13.099702 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.099714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.100639 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.100645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.102429 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.102443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.103060 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.103066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.103266 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.103801 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:13.103823 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:13.106900 (+ 3077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 02:13:13.107252 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:13.098666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:47293 (local address 127.24.216.190:36003)
0929 02:13:13.098796 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:13.098800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:13.098813 (+ 13us) server_negotiation.cc:407] Connection header received
0929 02:13:13.098846 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:13.098849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:13.098903 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:13.098974 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:13.099841 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.100500 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.102556 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.102932 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.103022 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.107053 (+ 4031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:47293: BlockingRecv error: recv got EOF from 127.24.216.129:47293 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 02:13:13.107632 1930 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:13:13.404809 1934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:13.405061 1934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:13.405187 1934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:13.412377 1934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:13.412492 1934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:13.412648 1934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:13:13.422526 1934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.130.keytab
--principal=oryx/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:44365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=46203
--tserver_master_addrs=127.24.216.190:36003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:13.423744 1934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:13.425040 1934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:13.437657 1942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:13.437728 1939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:13.437764 1940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:13.438503 1934 server_base.cc:1034] running on GCE node
I20260929 02:13:13.439059 1934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:13.439854 1934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:13.441046 1934 hybrid_clock.cc:648] HybridClock initialized: now 1790647993441011 us; error 34 us; skew 500 ppm
I20260929 02:13:13.441298 1934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647993, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:13.444201 1934 init.cc:377] Logged in from keytab as oryx/127.24.216.130@KRBTEST.COM (short username oryx)
I20260929 02:13:13.445447 1934 webserver.cc:460] Webserver started at http://127.24.216.130:46203/ using document root <none> and password file <none>
I20260929 02:13:13.446058 1934 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:13.446218 1934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:13.451180 1934 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:13:13.453774 1949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:13.455098 1934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:13:13.455259 1934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "7f35f9b5f7e8468498d7c5908d97a144"
format_stamp: "Formatted at 2026-09-29 02:13:10 on dist-test-slave-9k0h"
server_key: "fd61bb09140d811e5a166e22dff4d3a1"
server_key_iv: "bee7a5c14221f640805e9ab73d235cb2"
server_key_version: "encryptionkey@0"
I20260929 02:13:13.456166 1934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:13.483970 1934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:13.488106 1934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:13.488377 1934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:13.489341 1934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:13.490943 1934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:13.491062 1934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:13.491174 1934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:13.491250 1934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:13.523327 1934 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:44365
I20260929 02:13:13.523413 2061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:44365 every 8 connection(s)
I20260929 02:13:13.524740 1934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:13:13.532373 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 1934
I20260929 02:13:13.533062 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:35541
--local_ip_for_outbound_sockets=127.24.216.131
--tserver_master_addrs=127.24.216.190:36003
--webserver_port=36933
--webserver_interface=127.24.216.131
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.216.131
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:13:13.541710 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:13.527347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:33475 (local address 127.24.216.190:36003)
0929 02:13:13.527510 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:13.527514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:13.528677 (+ 1163us) server_negotiation.cc:407] Connection header received
0929 02:13:13.530447 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:13.530451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:13.530513 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:13.530599 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:13.531959 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.533080 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.534682 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.535435 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.535575 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.541469 (+ 5894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:33475: BlockingRecv error: recv got EOF from 127.24.216.130:33475 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:13:13.542474 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:13.527862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:33475)
0929 02:13:13.528515 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:13.528562 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:13.530134 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:13.530788 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:13.530796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:13.531264 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:13.531800 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.531815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.533258 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.533266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.534535 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.534546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.535646 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.535655 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.535933 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.536692 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:13.536721 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:13.541316 (+ 4595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":252,"threads_started":1}
W20260929 02:13:13.543084 2062 heartbeater.cc:644] Failed to heartbeat to 127.24.216.190:36003 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.216.190:36003: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:13.550879 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:13.543662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:37263)
0929 02:13:13.543822 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:13.543842 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:13.543938 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:13.544266 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:13.544270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:13.544586 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:13.544708 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.544715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.545661 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.545667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.546588 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.546598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.547268 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.547273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.547442 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.548009 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:13.548025 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:13.550615 (+ 2590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 02:13:13.551043 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:13.543742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:37263 (local address 127.24.216.190:36003)
0929 02:13:13.543908 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:13.543913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:13.543929 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:13:13.544078 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:13.544083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:13.544150 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:13.544245 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:13.544846 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.545529 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.546722 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.547120 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.547257 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.550796 (+ 3539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:37263: BlockingRecv error: recv got EOF from 127.24.216.130:37263 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:13.558909 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:13.551584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:55035)
0929 02:13:13.551740 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:13.551760 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:13.551850 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:13.552159 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:13.552164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:13.552447 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:13.552572 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.552579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.553607 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.553614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.554612 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.554622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.555260 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.555266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.555437 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.556018 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:13.556034 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:13.558619 (+ 2585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 02:13:13.559019 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:13.551667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:55035 (local address 127.24.216.190:36003)
0929 02:13:13.551836 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:13.551841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:13.551857 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:13:13.551974 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:13.551979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:13.552043 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:13.552139 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:13.552699 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.553478 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.554745 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.555123 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.555251 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.558767 (+ 3516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:55035: BlockingRecv error: recv got EOF from 127.24.216.130:55035 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 02:13:13.559207 2062 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:13:13.853746 2065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:13.854054 2065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:13.854244 2065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:13.861487 2065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:13.861630 2065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:13.861809 2065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:13:13.871446 2065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/krb5kdc/oryx_127.24.216.131.keytab
--principal=oryx/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:35541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=36933
--tserver_master_addrs=127.24.216.190:36003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:13.872780 2065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:13.874059 2065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:13.886441 2070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:13.886665 2071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:13.886665 2073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:13.887629 2065 server_base.cc:1034] running on GCE node
I20260929 02:13:13.888186 2065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:13.889047 2065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:13.890249 2065 hybrid_clock.cc:648] HybridClock initialized: now 1790647993890197 us; error 53 us; skew 500 ppm
I20260929 02:13:13.890544 2065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647993, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:13.893586 2065 init.cc:377] Logged in from keytab as oryx/127.24.216.131@KRBTEST.COM (short username oryx)
I20260929 02:13:13.894893 2065 webserver.cc:460] Webserver started at http://127.24.216.131:36933/ using document root <none> and password file <none>
I20260929 02:13:13.895500 2065 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:13.895635 2065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:13.900691 2065 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:13:13.903172 2080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:13.904512 2065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:13:13.904682 2065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "e5b9be886df44e0fa32086e67e5e3630"
format_stamp: "Formatted at 2026-09-29 02:13:11 on dist-test-slave-9k0h"
server_key: "eb3dcfe003ffeb75691b875e4cebbe79"
server_key_iv: "28240e99ddf8600e1367ceabe66f1db1"
server_key_version: "encryptionkey@0"
I20260929 02:13:13.905582 2065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:13.920815 2065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:13.924484 2065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:13.924746 2065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:13.925631 2065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:13.927004 2065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:13.927131 2065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:13.927255 2065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:13.927349 2065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:13.959659 2065 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:35541
I20260929 02:13:13.959708 2192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:35541 every 8 connection(s)
I20260929 02:13:13.961333 2065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:13:13.966898 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 2065
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:13:13.979687 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:13.963866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:56501 (local address 127.24.216.190:36003)
0929 02:13:13.964044 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:13.964049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:13.965128 (+ 1079us) server_negotiation.cc:407] Connection header received
0929 02:13:13.967149 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:13.967155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:13.967254 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:13.967363 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:13.969678 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.970684 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.972835 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.973590 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.973736 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.979457 (+ 5721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:56501: BlockingRecv error: recv got EOF from 127.24.216.131:56501 (error 108)
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":62,"spinlock_wait_cycles":1280}
I20260929 02:13:13.980563 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:13.964260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:56501)
0929 02:13:13.964959 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:13.965011 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:13.966876 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:13.967617 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:13.967630 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:13.968925 (+ 1295us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:13.969511 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.969527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.970829 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.970833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.972662 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.972679 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.973789 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.973796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.974091 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.974684 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:13.974712 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:13.979288 (+ 4576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":279,"threads_started":1}
W20260929 02:13:13.981189 2193 heartbeater.cc:644] Failed to heartbeat to 127.24.216.190:36003 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.216.190:36003: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
I20260929 02:13:13.984400 2197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:13.967880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.24.216.190:36003)
0929 02:13:13.968337 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:13.968343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:13.968360 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:13:13.968514 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:13.968518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:13.968592 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:13.968702 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:13.970136 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.971159 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.972877 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.973574 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.973742 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.976929 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:13.976939 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:13.976958 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:13.977012 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:13.980631 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:13.981922 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:13.981932 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:13.981939 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:13.982194 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:13.982560 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:13.982567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:13.982570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:13.983013 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:13.983344 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:13.983777 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 02:13:13.984122 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":361,"spinlock_wait_cycles":4352,"thread_start_us":258,"threads_started":1}
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:13.989442 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:13.981785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:32813)
0929 02:13:13.981945 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:13.981962 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:13.982045 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:13.982431 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:13.982435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:13.982744 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:13.982859 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.982866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.983844 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.983849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.984973 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.984985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.985883 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.985888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.986004 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.986558 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:13.986574 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:13.989268 (+ 2694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 02:13:13.989643 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:13.982016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:32813 (local address 127.24.216.190:36003)
0929 02:13:13.982217 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:13.982221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:13.982234 (+ 13us) server_negotiation.cc:407] Connection header received
0929 02:13:13.982268 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:13.982271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:13.982315 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:13.982412 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:13.982995 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.983712 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.985104 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.985742 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.985860 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.989417 (+ 3557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:32813: BlockingRecv error: recv got EOF from 127.24.216.131:32813 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:13 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:13.997051 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:13.990056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:49015)
0929 02:13:13.990251 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:13.990266 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:13.990365 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:13.990686 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:13.990691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:13.990982 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:13.991106 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.991113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.992040 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.992047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.992945 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:13.992954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.993587 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.993593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:13.993784 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.994378 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:13.994393 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:13.996870 (+ 2477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 02:13:13.997269 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:13.990216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:49015 (local address 127.24.216.190:36003)
0929 02:13:13.990460 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:13.990465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:13.990485 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:13:13.990516 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:13.990519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:13.990572 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:13.990667 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:13.991240 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.991914 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.993078 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:13.993453 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:13.993580 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:13.997045 (+ 3465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:49015: BlockingRecv error: recv got EOF from 127.24.216.131:49015 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
W20260929 02:13:13.997402 2193 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 02:13:14 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:14 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:14.116029 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:14.108200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:38091)
0929 02:13:14.108701 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:14.108723 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:14.108846 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:14.109120 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:14.109122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:14.109603 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:14.109830 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:14.109843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:14.110811 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:14.110815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:14.111769 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:14.111780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:14.112407 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:14.112411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:14.112557 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:14.113093 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:14.113109 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:14.115746 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":267,"threads_started":1}
W20260929 02:13:14.116202 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:14.108225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:38091 (local address 127.24.216.190:36003)
0929 02:13:14.108406 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:14.108414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:14.108808 (+ 394us) server_negotiation.cc:407] Connection header received
0929 02:13:14.108938 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:14.108942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:14.108999 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:14.109078 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:14.109964 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:14.110691 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:14.111886 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:14.112262 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:14.112357 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:14.115985 (+ 3628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:38091: BlockingRecv error: recv got EOF from 127.24.216.129:38091 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 02:13:14 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:14 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:14.568104 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:14.559881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:47755)
0929 02:13:14.560333 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:14.560349 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:14.560461 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:14.560743 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:14.560746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:14.561223 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:14.561425 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:14.561434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:14.562552 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:14.562559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:14.563675 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:14.563686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:14.564333 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:14.564337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:14.564449 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:14.565106 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:14.565123 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:14.567891 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":231,"threads_started":1}
W20260929 02:13:14.568269 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:14.559934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:47755 (local address 127.24.216.190:36003)
0929 02:13:14.560175 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:14.560181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:14.560430 (+ 249us) server_negotiation.cc:407] Connection header received
0929 02:13:14.560560 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:14.560563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:14.560622 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:14.560703 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:14.561562 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:14.562401 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:14.563792 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:14.564181 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:14.564273 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:14.568056 (+ 3783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:47755: BlockingRecv error: recv got EOF from 127.24.216.130:47755 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 02:13:14 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:14 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:15.005748 2200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:14.997988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:48859)
0929 02:13:14.998464 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:14.998479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:14.998631 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:14.999039 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:14.999044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:14.999548 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:14.999727 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:14.999734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:15.000651 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:15.000654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:15.001680 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:15.001690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:15.002365 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:15.002370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:15.002485 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:15.003017 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:15.003033 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:15.005535 (+ 2502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":251,"threads_started":1}
W20260929 02:13:15.005937 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:14.998049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:48859 (local address 127.24.216.190:36003)
0929 02:13:14.998262 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:14.998271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:14.998580 (+ 309us) server_negotiation.cc:407] Connection header received
0929 02:13:14.998769 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:14.998774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:14.998879 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:14.998975 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:14.999845 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:15.000533 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:15.001799 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:15.002209 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:15.002309 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:15.005705 (+ 3396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:48859: BlockingRecv error: recv got EOF from 127.24.216.131:48859 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 02:13:15 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:15 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:15.125150 2201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:15.116856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:52673)
0929 02:13:15.117386 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:15.117411 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:15.117581 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:15.117983 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:15.117988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:15.118407 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:15.118627 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:15.118635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:15.119667 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:15.119674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:15.120585 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:15.120597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:15.121330 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:15.121336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:15.121505 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:15.122036 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:15.122052 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:15.124801 (+ 2749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":271,"threads_started":1}
W20260929 02:13:15.125241 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:15.116925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:52673 (local address 127.24.216.190:36003)
0929 02:13:15.117168 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:15.117175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:15.117499 (+ 324us) server_negotiation.cc:407] Connection header received
0929 02:13:15.117717 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:15.117722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:15.117800 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:15.117940 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:15.118759 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:15.119526 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:15.120726 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:15.121161 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:15.121300 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:15.125002 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:52673: BlockingRecv error: recv got EOF from 127.24.216.129:52673 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 29 02:13:15 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:15 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:15.581660 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:15.568950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:55911)
0929 02:13:15.569470 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:15.569485 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:15.569604 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:15.569968 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:15.569972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:15.570476 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:15.570655 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:15.570664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:15.571628 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:15.571633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:15.572562 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:15.572573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:15.573196 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:15.573201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:15.573305 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:15.578475 (+ 5170us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:15.578500 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:15.581431 (+ 2931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":300,"threads_started":1}
W20260929 02:13:15.581795 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:15.569040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:55911 (local address 127.24.216.190:36003)
0929 02:13:15.569205 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:15.569209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:15.569584 (+ 375us) server_negotiation.cc:407] Connection header received
0929 02:13:15.569736 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:15.569739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:15.569799 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:15.569888 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:15.570794 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:15.571511 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:15.572679 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:15.573066 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:15.573152 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:15.581606 (+ 8454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:55911: BlockingRecv error: recv got EOF from 127.24.216.130:55911 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 02:13:16 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:16 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:16.014804 2203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:16.006777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:35927)
0929 02:13:16.007327 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:16.007343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:16.007501 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:16.007922 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:16.007927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:16.008449 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:16.008654 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:16.008668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:16.009753 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:16.009756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:16.010742 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:16.010753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:16.011385 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:16.011389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:16.011488 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:16.012007 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:16.012022 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:16.014588 (+ 2566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":280,"threads_started":1}
W20260929 02:13:16.014982 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:16.006888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:35927 (local address 127.24.216.190:36003)
0929 02:13:16.007129 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:16.007135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:16.007435 (+ 300us) server_negotiation.cc:407] Connection header received
0929 02:13:16.007654 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:16.007659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:16.007762 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:16.007866 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:16.008797 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:16.009629 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:16.010855 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:16.011251 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:16.011351 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:16.014751 (+ 3400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:35927: BlockingRecv error: recv got EOF from 127.24.216.131:35927 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 02:13:16 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:16 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:16.133785 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:16.126223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:33183)
0929 02:13:16.126717 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:16.126737 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:16.126844 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:16.127105 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:16.127107 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:16.127427 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:16.127579 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:16.127587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:16.128524 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:16.128527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:16.129419 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:16.129429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:16.130035 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:16.130039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:16.130285 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:16.130885 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:16.130902 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:16.133520 (+ 2618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":264,"threads_started":1}
W20260929 02:13:16.134033 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:16.126288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:33183 (local address 127.24.216.190:36003)
0929 02:13:16.126437 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:16.126441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:16.126808 (+ 367us) server_negotiation.cc:407] Connection header received
0929 02:13:16.126931 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:16.126934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:16.126989 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:16.127064 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:16.127694 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:16.128414 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:16.129555 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:16.129924 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:16.130009 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:16.133703 (+ 3694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:33183: BlockingRecv error: recv got EOF from 127.24.216.129:33183 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 02:13:16 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:16 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:16.590219 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:16.582592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:34141)
0929 02:13:16.583027 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:16.583044 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:16.583152 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:16.583437 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:16.583440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:16.583774 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:16.583928 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:16.583936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:16.584893 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:16.584896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:16.585773 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:16.585782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:16.586424 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:16.586428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:16.586543 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:16.587154 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:16.587171 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:16.589908 (+ 2737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":232,"threads_started":1}
W20260929 02:13:16.590371 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:16.582631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:34141 (local address 127.24.216.190:36003)
0929 02:13:16.582904 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:16.582911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:16.583119 (+ 208us) server_negotiation.cc:407] Connection header received
0929 02:13:16.583255 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:16.583258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:16.583318 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:16.583397 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:16.584045 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:16.584770 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:16.585883 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:16.586316 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:16.586402 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:16.590088 (+ 3686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:34141: BlockingRecv error: recv got EOF from 127.24.216.130:34141 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 29 02:13:17 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:17 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:17.023295 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:17.015692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:50299)
0929 02:13:17.016181 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:17.016196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:17.016300 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:17.016584 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:17.016587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:17.016900 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:17.017081 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:17.017093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:17.018064 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:17.018067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:17.019108 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:17.019119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:17.019750 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:17.019754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:17.019877 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:17.020425 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:17.020441 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:17.023064 (+ 2623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":258,"threads_started":1}
W20260929 02:13:17.023518 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:17.015774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:50299 (local address 127.24.216.190:36003)
0929 02:13:17.015935 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:17.015939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:17.016262 (+ 323us) server_negotiation.cc:407] Connection header received
0929 02:13:17.016393 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:17.016396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:17.016473 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:17.016551 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:17.017249 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:17.017945 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:17.019220 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:17.019630 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:17.019721 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:17.023265 (+ 3544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:50299: BlockingRecv error: recv got EOF from 127.24.216.131:50299 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 02:13:17 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:17 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:17.142576 2207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:17.134607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:33167)
0929 02:13:17.135094 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:17.135112 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:17.135267 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:17.135627 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:17.135631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:17.135992 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:17.136129 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:17.136137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:17.137106 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:17.137114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:17.138214 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:17.138231 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:17.138915 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:17.138920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:17.139095 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:17.139655 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:17.139671 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:17.142239 (+ 2568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":260,"threads_started":1}
W20260929 02:13:17.142702 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:17.134661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:33167 (local address 127.24.216.190:36003)
0929 02:13:17.134915 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:17.134923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:17.135197 (+ 274us) server_negotiation.cc:407] Connection header received
0929 02:13:17.135395 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:17.135400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:17.135470 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:17.135573 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:17.136268 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:17.136979 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:17.138355 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:17.138755 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:17.138899 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:17.142453 (+ 3554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:33167: BlockingRecv error: recv got EOF from 127.24.216.129:33167 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 02:13:17 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:17 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:17.599545 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:17.591311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:35419)
0929 02:13:17.591845 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:17.591860 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:17.592035 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:17.592368 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:17.592373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:17.592735 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:17.592872 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:17.592880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:17.593864 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:17.593871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:17.594897 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:17.594908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:17.595632 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:17.595637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:17.595814 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:17.596458 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:17.596475 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:17.599215 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":332,"threads_started":1}
W20260929 02:13:17.599618 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:17.591370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:35419 (local address 127.24.216.190:36003)
0929 02:13:17.591546 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:17.591553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:17.591964 (+ 411us) server_negotiation.cc:407] Connection header received
0929 02:13:17.592158 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:17.592163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:17.592237 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:17.592340 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:17.593009 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:17.593730 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:17.595034 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:17.595471 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:17.595614 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:17.599404 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:35419: BlockingRecv error: recv got EOF from 127.24.216.130:35419 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 02:13:18 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:18 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:18.031627 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:18.024140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:60433)
0929 02:13:18.024626 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:18.024642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:18.024751 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:18.025043 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:18.025046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:18.025386 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:18.025524 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:18.025532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:18.026521 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:18.026525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:18.027445 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:18.027456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:18.028032 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:18.028036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:18.028142 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:18.028675 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:18.028691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:18.031419 (+ 2728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":275,"threads_started":1}
W20260929 02:13:18.031821 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:18.024217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:60433 (local address 127.24.216.190:36003)
0929 02:13:18.024392 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:18.024396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:18.024710 (+ 314us) server_negotiation.cc:407] Connection header received
0929 02:13:18.024854 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:18.024857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:18.024926 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:18.025003 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:18.025652 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:18.026417 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:18.027552 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:18.027927 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:18.028009 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:18.031586 (+ 3577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:60433: BlockingRecv error: recv got EOF from 127.24.216.131:60433 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 02:13:18 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:18 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:18.151188 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:18.143724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:58677)
0929 02:13:18.144242 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:18.144262 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:18.144373 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:18.144631 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:18.144634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:18.145009 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:18.145160 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:18.145167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:18.146092 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:18.146096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:18.147043 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:18.147054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:18.147684 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:18.147687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:18.147797 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:18.148338 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:18.148354 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:18.150955 (+ 2601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":267,"threads_started":1}
W20260929 02:13:18.151364 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:18.143917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:58677 (local address 127.24.216.190:36003)
0929 02:13:18.144098 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:18.144103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:18.144325 (+ 222us) server_negotiation.cc:407] Connection header received
0929 02:13:18.144461 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:18.144465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:18.144519 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:18.144595 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:18.145278 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:18.145982 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:18.147149 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:18.147574 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:18.147660 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:18.151143 (+ 3483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:58677: BlockingRecv error: recv got EOF from 127.24.216.129:58677 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 02:13:18 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:18 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:18.608358 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:18.600508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:33687)
0929 02:13:18.601021 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:18.601037 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:18.601141 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:18.601403 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:18.601406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:18.601742 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:18.601917 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:18.601925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:18.603005 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:18.603009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:18.603951 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:18.603964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:18.604635 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:18.604639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:18.604771 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:18.605308 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:18.605327 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:18.607941 (+ 2614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":269,"threads_started":1}
W20260929 02:13:18.608471 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:18.600648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:33687 (local address 127.24.216.190:36003)
0929 02:13:18.600896 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:18.600902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:18.601114 (+ 212us) server_negotiation.cc:407] Connection header received
0929 02:13:18.601223 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:18.601226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:18.601280 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:18.601358 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:18.602027 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:18.602855 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:18.604117 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:18.604508 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:18.604606 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:18.608192 (+ 3586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:33687: BlockingRecv error: recv got EOF from 127.24.216.130:33687 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 02:13:19 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:19 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:19.040688 2212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:19.032514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:50527)
0929 02:13:19.033040 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:19.033057 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:19.033202 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:19.033555 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:19.033559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:19.034020 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:19.034207 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:19.034216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:19.035215 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:19.035223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:19.036148 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:19.036158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:19.036840 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:19.036846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:19.037025 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:19.037627 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:19.037654 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:19.040332 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":253,"threads_started":1}
W20260929 02:13:19.040795 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:19.032718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:50527 (local address 127.24.216.190:36003)
0929 02:13:19.032939 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:19.032946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:19.033141 (+ 195us) server_negotiation.cc:407] Connection header received
0929 02:13:19.033358 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:19.033363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:19.033431 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:19.033527 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:19.034365 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:19.035076 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:19.036288 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:19.036701 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:19.036839 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:19.040516 (+ 3677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:50527: BlockingRecv error: recv got EOF from 127.24.216.131:50527 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 02:13:19 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:19 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:19.160283 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:19.152235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:60507)
0929 02:13:19.152677 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:19.152696 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:19.152803 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:19.153098 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:19.153101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:19.153517 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:19.153652 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:19.153660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:19.154735 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:19.154739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:19.155585 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:19.155595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:19.156211 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:19.156215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:19.156336 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:19.156878 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:19.156894 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:19.160044 (+ 3150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":232,"threads_started":1}
W20260929 02:13:19.160597 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:19.152243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:60507 (local address 127.24.216.190:36003)
0929 02:13:19.152448 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:19.152456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:19.152820 (+ 364us) server_negotiation.cc:407] Connection header received
0929 02:13:19.152875 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:19.152878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:19.152935 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:19.153152 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:19.153765 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:19.154619 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:19.155693 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:19.156092 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:19.156179 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:19.160349 (+ 4170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:60507: BlockingRecv error: recv got EOF from 127.24.216.129:60507 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 02:13:19 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:19 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:19.617479 2214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:19.609367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:40625)
0929 02:13:19.609853 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:19.609876 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:19.609979 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:19.610304 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:19.610307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:19.610630 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:19.610810 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:19.610818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:19.611811 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:19.611815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:19.612654 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:19.612664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:19.613312 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:19.613316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:19.613430 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:19.614068 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:19.614085 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:19.617148 (+ 3063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":254,"threads_started":1}
W20260929 02:13:19.617615 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:19.609458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:40625 (local address 127.24.216.190:36003)
0929 02:13:19.609683 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:19.609689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:19.609945 (+ 256us) server_negotiation.cc:407] Connection header received
0929 02:13:19.610082 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:19.610086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:19.610189 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:19.610275 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:19.610941 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:19.611698 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:19.612794 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:19.613184 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:19.613269 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:19.617372 (+ 4103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:40625: BlockingRecv error: recv got EOF from 127.24.216.130:40625 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 02:13:20 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:20 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:20.049602 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:20.041706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:53571)
0929 02:13:20.042245 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:20.042260 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:20.042393 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:20.042663 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:20.042666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:20.043062 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:20.043191 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:20.043199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:20.044332 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:20.044336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:20.045323 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:20.045334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:20.046137 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:20.046143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:20.046332 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:20.046968 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:20.046984 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:20.049388 (+ 2404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":311,"threads_started":1}
W20260929 02:13:20.049799 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:20.041740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:53571 (local address 127.24.216.190:36003)
0929 02:13:20.041977 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:20.041987 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:20.042356 (+ 369us) server_negotiation.cc:407] Connection header received
0929 02:13:20.042488 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:20.042491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:20.042549 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:20.042625 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:20.043314 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:20.044222 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:20.045457 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:20.045995 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:20.046082 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:20.049575 (+ 3493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:53571: BlockingRecv error: recv got EOF from 127.24.216.131:53571 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 02:13:20 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:20 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:20.169324 2216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:20.161174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:43567)
0929 02:13:20.161713 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:20.161731 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:20.161889 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:20.162267 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:20.162273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:20.162673 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:20.162830 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:20.162838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:20.163824 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:20.163829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:20.164744 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:20.164755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:20.165456 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:20.165462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:20.165630 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:20.166272 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:20.166298 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:20.168974 (+ 2676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":257,"threads_started":1}
W20260929 02:13:20.169418 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:20.161252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:43567 (local address 127.24.216.190:36003)
0929 02:13:20.161435 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:20.161444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:20.161821 (+ 377us) server_negotiation.cc:407] Connection header received
0929 02:13:20.162009 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:20.162014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:20.162086 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:20.162209 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:20.162966 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:20.163672 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:20.164885 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:20.165287 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:20.165441 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:20.169172 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:43567: BlockingRecv error: recv got EOF from 127.24.216.129:43567 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 02:13:20 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:20 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:20.632048 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:20.618622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:58205)
0929 02:13:20.619168 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:20.619187 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:20.619313 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:20.619676 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:20.619681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:20.620103 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:20.620257 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:20.620265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:20.621287 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:20.621294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:20.622228 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:20.622238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:20.622891 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:20.622896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:20.623059 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:20.628669 (+ 5610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:20.628691 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:20.631807 (+ 3116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":222,"threads_started":1}
W20260929 02:13:20.632339 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:20.618649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:58205 (local address 127.24.216.190:36003)
0929 02:13:20.618785 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:20.618789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:20.619251 (+ 462us) server_negotiation.cc:407] Connection header received
0929 02:13:20.619481 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:20.619490 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:20.619547 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:20.619646 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:20.620394 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:20.621148 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:20.622362 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:20.622751 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:20.622879 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:20.632042 (+ 9163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:58205: BlockingRecv error: recv got EOF from 127.24.216.130:58205 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 02:13:21 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:21 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:21.058895 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:21.050561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:44003)
0929 02:13:21.051119 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:21.051136 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:21.051285 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:21.051613 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:21.051620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:21.052037 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:21.052194 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:21.052203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:21.053655 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:21.053661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:21.054705 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:21.054717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:21.055388 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:21.055392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:21.055538 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:21.056071 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:21.056088 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:21.058644 (+ 2556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":255,"threads_started":1}
W20260929 02:13:21.059001 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:21.050639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:44003 (local address 127.24.216.190:36003)
0929 02:13:21.050838 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:21.050847 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:21.051215 (+ 368us) server_negotiation.cc:407] Connection header received
0929 02:13:21.051417 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:21.051421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:21.051477 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:21.051573 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:21.052346 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:21.053478 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:21.054823 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:21.055248 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:21.055345 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:21.058818 (+ 3473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:44003: BlockingRecv error: recv got EOF from 127.24.216.131:44003 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 02:13:21 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:21 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:21.178208 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:21.170337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:51457)
0929 02:13:21.170889 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:21.170908 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:21.171021 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:21.171290 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:21.171293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:21.171665 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:21.171810 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:21.171819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:21.172754 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:21.172758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:21.173650 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:21.173660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:21.174352 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:21.174356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:21.174459 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:21.175097 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:21.175113 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:21.177827 (+ 2714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":289,"threads_started":1}
W20260929 02:13:21.178356 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:21.170483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:51457 (local address 127.24.216.190:36003)
0929 02:13:21.170669 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:21.170673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:21.170983 (+ 310us) server_negotiation.cc:407] Connection header received
0929 02:13:21.171106 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:21.171109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:21.171168 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:21.171244 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:21.171926 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:21.172643 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:21.173764 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:21.174211 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:21.174297 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:21.178047 (+ 3750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:51457: BlockingRecv error: recv got EOF from 127.24.216.129:51457 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 02:13:21 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:21 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:21.641678 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:21.632970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:41861)
0929 02:13:21.633440 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:21.633456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:21.633568 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:21.633857 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:21.633860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:21.634302 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:21.634459 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:21.634467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:21.635697 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:21.635702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:21.637101 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:21.637112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:21.637778 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:21.637782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:21.637882 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:21.638617 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:21.638634 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:21.641365 (+ 2731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
W20260929 02:13:21.641733 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:21.633099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:41861 (local address 127.24.216.190:36003)
0929 02:13:21.633318 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:21.633324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:21.633533 (+ 209us) server_negotiation.cc:407] Connection header received
0929 02:13:21.633678 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:21.633681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:21.633737 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:21.633814 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:21.634566 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:21.635545 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:21.637220 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:21.637636 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:21.637726 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:21.641527 (+ 3801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:41861: BlockingRecv error: recv got EOF from 127.24.216.130:41861 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 29 02:13:22 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:22 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:22.068070 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:22.059936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:37535)
0929 02:13:22.060448 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:22.060463 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:22.060622 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:22.060969 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:22.060972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:22.061341 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:22.061480 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:22.061488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:22.062510 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:22.062514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:22.063394 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:22.063404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:22.064058 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:22.064062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:22.064171 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:22.064800 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:22.064817 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:22.067852 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":266,"threads_started":1}
W20260929 02:13:22.068259 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:22.060058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:37535 (local address 127.24.216.190:36003)
0929 02:13:22.060292 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:22.060298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:22.060586 (+ 288us) server_negotiation.cc:407] Connection header received
0929 02:13:22.060768 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:22.060774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:22.060851 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:22.060927 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:22.061592 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:22.062395 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:22.063513 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:22.063921 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:22.064007 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:22.068007 (+ 4000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:37535: BlockingRecv error: recv got EOF from 127.24.216.131:37535 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 02:13:22 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:22 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:22.187004 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:22.179105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:36627)
0929 02:13:22.179594 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:22.179613 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:22.179722 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:22.179985 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:22.179988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:22.180344 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:22.180494 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:22.180502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:22.181470 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:22.181474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:22.182415 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:22.182425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:22.183052 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:22.183055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:22.183197 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:22.183888 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:22.183912 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:22.186770 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":277,"threads_started":1}
W20260929 02:13:22.187124 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:22.179178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:36627 (local address 127.24.216.190:36003)
0929 02:13:22.179473 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:22.179479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:22.179712 (+ 233us) server_negotiation.cc:407] Connection header received
0929 02:13:22.179813 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:22.179817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:22.179875 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:22.179945 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:22.180612 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:22.181364 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:22.182526 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:22.182937 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:22.183025 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:22.186932 (+ 3907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:36627: BlockingRecv error: recv got EOF from 127.24.216.129:36627 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 02:13:22 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:22 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:22.650192 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:22.642563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:45007)
0929 02:13:22.643025 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:22.643041 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:22.643163 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:22.643446 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:22.643449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:22.643789 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:22.643942 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:22.643950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:22.644899 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:22.644902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:22.645810 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:22.645820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:22.646553 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:22.646557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:22.646680 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:22.647212 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:22.647228 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:22.649918 (+ 2690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
W20260929 02:13:22.650398 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:22.642650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:45007 (local address 127.24.216.190:36003)
0929 02:13:22.642896 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:22.642902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:22.643131 (+ 229us) server_negotiation.cc:407] Connection header received
0929 02:13:22.643259 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:22.643263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:22.643319 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:22.643394 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:22.644068 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:22.644781 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:22.645989 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:22.646417 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:22.646515 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:22.650100 (+ 3585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:45007: BlockingRecv error: recv got EOF from 127.24.216.130:45007 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 29 02:13:23 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:23 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:23.076711 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:23.068883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:46867)
0929 02:13:23.069387 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:23.069402 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:23.069550 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:23.069879 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:23.069884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:23.070332 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:23.070482 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:23.070490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:23.071474 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:23.071479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:23.072420 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:23.072431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:23.073092 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:23.073098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:23.073266 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:23.073840 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:23.073857 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:23.076489 (+ 2632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":265,"threads_started":1}
W20260929 02:13:23.076855 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:23.068924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:46867 (local address 127.24.216.190:36003)
0929 02:13:23.069172 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:23.069180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:23.069481 (+ 301us) server_negotiation.cc:407] Connection header received
0929 02:13:23.069682 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:23.069686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:23.069747 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:23.069843 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:23.070613 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:23.071341 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:23.072558 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:23.072954 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:23.073083 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:23.076660 (+ 3577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:46867: BlockingRecv error: recv got EOF from 127.24.216.131:46867 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 02:13:23 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:23 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:23.196105 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:23.188000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:34715)
0929 02:13:23.188467 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:23.188486 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:23.188634 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:23.188969 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:23.188974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:23.189450 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:23.189592 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:23.189600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:23.190709 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:23.190717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:23.191658 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:23.191669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:23.192391 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:23.192396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:23.192560 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:23.193103 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:23.193128 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:23.195898 (+ 2770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":251,"threads_started":1}
W20260929 02:13:23.196466 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:23.188080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:34715 (local address 127.24.216.190:36003)
0929 02:13:23.188342 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:23.188348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:23.188570 (+ 222us) server_negotiation.cc:407] Connection header received
0929 02:13:23.188758 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:23.188763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:23.188845 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:23.188941 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:23.189735 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:23.190587 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:23.191794 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:23.192236 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:23.192378 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:23.196136 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:34715: BlockingRecv error: recv got EOF from 127.24.216.129:34715 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 02:13:23 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:23 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:23.659403 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:23.651225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:44499)
0929 02:13:23.651698 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:23.651713 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:23.651859 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:23.652193 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:23.652198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:23.652601 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:23.652723 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:23.652730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:23.653665 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:23.653668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:23.654583 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:23.654593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:23.655261 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:23.655266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:23.655427 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:23.655979 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:23.655995 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:23.659013 (+ 3018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":273,"threads_started":1}
W20260929 02:13:23.659539 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:23.651302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:44499 (local address 127.24.216.190:36003)
0929 02:13:23.651517 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:23.651526 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:23.651794 (+ 268us) server_negotiation.cc:407] Connection header received
0929 02:13:23.651981 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:23.651992 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:23.652050 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:23.652159 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:23.652858 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:23.653519 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:23.654720 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:23.655115 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:23.655234 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:23.659224 (+ 3990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:44499: BlockingRecv error: recv got EOF from 127.24.216.130:44499 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 02:13:24 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:24 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:24.085851 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:24.077627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:49731)
0929 02:13:24.078196 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:24.078215 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:24.078368 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:24.078704 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:24.078708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:24.079090 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:24.079224 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:24.079232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:24.080235 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:24.080241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:24.081169 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:24.081180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:24.081992 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:24.081997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:24.082227 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:24.082901 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:24.082917 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:24.085624 (+ 2707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":272,"threads_started":1}
W20260929 02:13:24.086006 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:24.077921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:49731 (local address 127.24.216.190:36003)
0929 02:13:24.078189 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:24.078195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:24.078304 (+ 109us) server_negotiation.cc:407] Connection header received
0929 02:13:24.078492 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:24.078498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:24.078576 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:24.078675 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:24.079389 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:24.080106 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:24.081302 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:24.081847 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:24.081981 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:24.085787 (+ 3806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:49731: BlockingRecv error: recv got EOF from 127.24.216.131:49731 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 29 02:13:24 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:24 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:24.205310 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:24.197003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:45905)
0929 02:13:24.197520 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:24.197545 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:24.197691 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:24.198087 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:24.198092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:24.198627 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:24.198856 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:24.198865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:24.200020 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:24.200024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:24.200889 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:24.200899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:24.201588 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:24.201592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:24.201704 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:24.202355 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:24.202370 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:24.205087 (+ 2717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":483,"threads_started":1}
W20260929 02:13:24.205627 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:24.197267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:45905 (local address 127.24.216.190:36003)
0929 02:13:24.197482 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:24.197488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:24.197680 (+ 192us) server_negotiation.cc:407] Connection header received
0929 02:13:24.197803 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:24.197807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:24.197902 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:24.198020 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:24.199051 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:24.199896 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:24.201006 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:24.201463 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:24.201565 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:24.205296 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:45905: BlockingRecv error: recv got EOF from 127.24.216.129:45905 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 02:13:24 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:24 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:24.668605 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:24.660254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:36417)
0929 02:13:24.660728 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:24.660743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:24.660897 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:24.661264 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:24.661269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:24.661674 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:24.661839 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:24.661847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:24.662899 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:24.662907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:24.663802 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:24.663812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:24.664558 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:24.664563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:24.664734 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:24.665376 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:24.665393 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:24.668274 (+ 2881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":286,"threads_started":1}
W20260929 02:13:24.668736 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:24.660316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:36417 (local address 127.24.216.190:36003)
0929 02:13:24.660527 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:24.660535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:24.660818 (+ 283us) server_negotiation.cc:407] Connection header received
0929 02:13:24.661062 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:24.661067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:24.661136 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:24.661235 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:24.661975 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:24.662774 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:24.663951 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:24.664382 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:24.664532 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:24.668499 (+ 3967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:36417: BlockingRecv error: recv got EOF from 127.24.216.130:36417 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 02:13:25 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:25 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:25.094226 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:25.086906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:38645)
0929 02:13:25.087315 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:25.087330 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:25.087432 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:25.087741 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:25.087744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:25.088109 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:25.088268 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:25.088275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:25.089161 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:25.089164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:25.090030 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:25.090040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:25.090676 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:25.090681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:25.090822 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:25.091380 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:25.091396 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:25.093991 (+ 2595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":224,"threads_started":1}
W20260929 02:13:25.094448 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:25.086950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:38645 (local address 127.24.216.190:36003)
0929 02:13:25.087166 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:25.087171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:25.087408 (+ 237us) server_negotiation.cc:407] Connection header received
0929 02:13:25.087522 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:25.087525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:25.087580 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:25.087655 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:25.088379 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:25.089061 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:25.090139 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:25.090536 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:25.090620 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:25.094189 (+ 3569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:38645: BlockingRecv error: recv got EOF from 127.24.216.131:38645 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 02:13:25 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:25 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:25.214658 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:25.206234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:36101)
0929 02:13:25.206766 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:25.206787 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:25.206912 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:25.207184 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:25.207187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:25.207548 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:25.207687 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:25.207694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:25.208657 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:25.208661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:25.209632 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:25.209642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:25.210458 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:25.210462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:25.210587 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:25.211218 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:25.211233 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:25.214305 (+ 3072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":307,"threads_started":1}
W20260929 02:13:25.214756 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:25.206355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:36101 (local address 127.24.216.190:36003)
0929 02:13:25.206533 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:25.206538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:25.206863 (+ 325us) server_negotiation.cc:407] Connection header received
0929 02:13:25.207003 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:25.207006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:25.207061 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:25.207140 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:25.207796 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:25.208520 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:25.209799 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:25.210323 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:25.210433 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:25.214528 (+ 4095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:36101: BlockingRecv error: recv got EOF from 127.24.216.129:36101 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 02:13:25 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:25 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:25.681648 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:25.669513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:43589)
0929 02:13:25.670033 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:25.670049 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:25.670210 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:25.670578 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:25.670582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:25.671014 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:25.671194 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:25.671202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:25.672152 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:25.672156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:25.673057 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:25.673067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:25.673752 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:25.673756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:25.673876 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:25.678751 (+ 4875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:25.678768 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:25.681373 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":304,"threads_started":1}
W20260929 02:13:25.681861 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:25.669560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:43589 (local address 127.24.216.190:36003)
0929 02:13:25.669776 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:25.669783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:25.670133 (+ 350us) server_negotiation.cc:407] Connection header received
0929 02:13:25.670342 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:25.670349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:25.670431 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:25.670505 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:25.671324 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:25.672035 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:25.673168 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:25.673625 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:25.673720 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:25.681616 (+ 7896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:43589: BlockingRecv error: recv got EOF from 127.24.216.130:43589 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 02:13:26 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:26 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:26.102716 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:26.095082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:33985)
0929 02:13:26.095552 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:26.095567 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:26.095680 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:26.095941 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:26.095944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:26.096303 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:26.096441 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:26.096448 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:26.097380 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:26.097383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:26.098263 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:26.098273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:26.098894 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:26.098898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:26.099012 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:26.099587 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:26.099603 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:26.102391 (+ 2788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":258,"threads_started":1}
W20260929 02:13:26.102804 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:26.095139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:33985 (local address 127.24.216.190:36003)
0929 02:13:26.095296 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:26.095300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:26.095636 (+ 336us) server_negotiation.cc:407] Connection header received
0929 02:13:26.095772 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:26.095776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:26.095832 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:26.095907 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:26.096549 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:26.097264 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:26.098386 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:26.098776 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:26.098867 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:26.102572 (+ 3705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:33985: BlockingRecv error: recv got EOF from 127.24.216.131:33985 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 02:13:26 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:26 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:26.223409 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:26.215433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:50949)
0929 02:13:26.215948 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:26.215966 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:26.216068 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:26.216328 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:26.216331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:26.216661 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:26.216800 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:26.216808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:26.217733 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:26.217737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:26.218748 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:26.218759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:26.219487 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:26.219492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:26.219643 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:26.220347 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:26.220363 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:26.223187 (+ 2824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":265,"threads_started":1}
W20260929 02:13:26.223572 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:26.215500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:50949 (local address 127.24.216.190:36003)
0929 02:13:26.215662 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:26.215666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:26.216028 (+ 362us) server_negotiation.cc:407] Connection header received
0929 02:13:26.216154 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:26.216157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:26.216217 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:26.216294 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:26.216916 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:26.217631 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:26.218866 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:26.219289 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:26.219452 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:26.223363 (+ 3911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:50949: BlockingRecv error: recv got EOF from 127.24.216.129:50949 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 02:13:26 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:26 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:26.690958 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:26.682758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:33457)
0929 02:13:26.683254 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:26.683270 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:26.683391 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:26.683662 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:26.683665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:26.684017 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:26.684180 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:26.684188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:26.685463 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:26.685470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:26.686477 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:26.686487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:26.687219 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:26.687225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:26.687387 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:26.687934 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:26.687949 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:26.690724 (+ 2775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":267,"threads_started":1}
W20260929 02:13:26.691104 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:26.682890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:33457 (local address 127.24.216.190:36003)
0929 02:13:26.683119 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:26.683124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:26.683348 (+ 224us) server_negotiation.cc:407] Connection header received
0929 02:13:26.683489 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:26.683492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:26.683549 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:26.683627 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:26.684321 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:26.685345 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:26.686621 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:26.687045 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:26.687188 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:26.690904 (+ 3716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:33457: BlockingRecv error: recv got EOF from 127.24.216.130:33457 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 02:13:27 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:27 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:27.111799 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:27.103780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:47165)
0929 02:13:27.104253 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:27.104267 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:27.104414 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:27.104725 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:27.104729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:27.105096 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:27.105232 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:27.105240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:27.106238 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:27.106251 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:27.107132 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:27.107142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:27.107934 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:27.107940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:27.108099 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:27.108736 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:27.108751 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:27.111459 (+ 2708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":241,"threads_started":1}
W20260929 02:13:27.111872 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:27.103856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:47165 (local address 127.24.216.190:36003)
0929 02:13:27.104068 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:27.104075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:27.104353 (+ 278us) server_negotiation.cc:407] Connection header received
0929 02:13:27.104537 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:27.104542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:27.104607 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:27.104698 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:27.105380 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:27.106080 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:27.107267 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:27.107758 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:27.107904 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:27.111651 (+ 3747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:47165: BlockingRecv error: recv got EOF from 127.24.216.131:47165 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 02:13:27 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:27 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:27.232582 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:27.224292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:51163)
0929 02:13:27.225001 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:27.225021 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:27.225170 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:27.225541 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:27.225545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:27.226028 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:27.226264 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:27.226274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:27.227249 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:27.227255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:27.228258 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:27.228269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:27.228944 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:27.228949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:27.229114 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:27.229646 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:27.229662 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:27.232223 (+ 2561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":482,"threads_started":1}
W20260929 02:13:27.232650 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:27.224354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:51163 (local address 127.24.216.190:36003)
0929 02:13:27.224743 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:27.224752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:27.225096 (+ 344us) server_negotiation.cc:407] Connection header received
0929 02:13:27.225327 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:27.225333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:27.225412 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:27.225515 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:27.226390 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:27.227120 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:27.228391 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:27.228810 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:27.228936 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:27.232417 (+ 3481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:51163: BlockingRecv error: recv got EOF from 127.24.216.129:51163 (error 108)
Metrics: {"server-negotiator.queue_time_us":193}
Sep 29 02:13:27 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:27 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:27.699909 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:27.691996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:60275)
0929 02:13:27.692463 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:27.692483 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:27.692606 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:27.692996 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:27.692999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:27.693370 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:27.693536 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:27.693543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:27.694558 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:27.694561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:27.695421 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:27.695430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:27.696334 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:27.696338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:27.696446 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:27.697043 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:27.697059 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:27.699680 (+ 2621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":237,"threads_started":1}
W20260929 02:13:27.700098 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:27.692269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:60275 (local address 127.24.216.190:36003)
0929 02:13:27.692441 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:27.692446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:27.692541 (+ 95us) server_negotiation.cc:407] Connection header received
0929 02:13:27.692766 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:27.692773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:27.692845 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:27.692932 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:27.693645 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:27.694445 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:27.695567 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:27.696203 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:27.696314 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:27.699890 (+ 3576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:60275: BlockingRecv error: recv got EOF from 127.24.216.130:60275 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 02:13:28 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:28 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.131@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:28.120537 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:28.112766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.131:33227)
0929 02:13:28.113191 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:28.113205 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:28.113308 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:28.113598 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:28.113601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:28.113925 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:28.114080 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:28.114087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:28.115146 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:28.115149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:28.116136 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:28.116147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:28.116803 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:28.116807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:28.116918 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:28.117500 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:28.117517 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:28.120276 (+ 2759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":218,"threads_started":1}
W20260929 02:13:28.120704 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:28.112777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:33227 (local address 127.24.216.190:36003)
0929 02:13:28.113011 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:28.113021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:28.113289 (+ 268us) server_negotiation.cc:407] Connection header received
0929 02:13:28.113414 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:28.113417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:28.113480 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:28.113566 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:28.114257 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:28.115010 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:28.116264 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:28.116689 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:28.116783 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:28.120475 (+ 3692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.131:33227: BlockingRecv error: recv got EOF from 127.24.216.131:33227 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 02:13:28 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:28 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.129@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:28.241662 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:28.233756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.129:39787)
0929 02:13:28.234298 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:28.234318 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:28.234417 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:28.234669 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:28.234672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:28.235074 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:28.235240 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:28.235257 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:28.236163 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:28.236166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:28.237028 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:28.237050 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:28.237706 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:28.237709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:28.237813 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:28.238488 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:28.238506 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:28.241411 (+ 2905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":283,"threads_started":1}
W20260929 02:13:28.241824 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:28.233839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:39787 (local address 127.24.216.190:36003)
0929 02:13:28.234017 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:28.234021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:28.234381 (+ 360us) server_negotiation.cc:407] Connection header received
0929 02:13:28.234493 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:28.234496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:28.234548 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:28.234621 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:28.235365 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:28.236062 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:28.237144 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:28.237577 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:28.237677 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:28.241589 (+ 3912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.129:39787: BlockingRecv error: recv got EOF from 127.24.216.129:39787 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 02:13:28 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:13:28 dist-test-slave-9k0h krb5kdc[1195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.216.130@KRBTEST.COM for oryx/127.24.216.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:13:28.709363 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:28.701022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36003 (local address 127.24.216.130:43123)
0929 02:13:28.701575 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:28.701592 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:28.701770 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:28.702234 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:28.702239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:28.702768 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:28.703000 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:28.703010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:28.703965 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:28.703968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:28.704913 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:28.704923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:28.705539 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:28.705544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:28.705645 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:28.706307 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:28.706323 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:28.709073 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.216.190:36003: Server oryx/127.24.216.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":304,"threads_started":1}
W20260929 02:13:28.709540 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:28.701072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:43123 (local address 127.24.216.190:36003)
0929 02:13:28.701342 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:28.701348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:28.701707 (+ 359us) server_negotiation.cc:407] Connection header received
0929 02:13:28.701937 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:28.701943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:28.702021 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:28.702123 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:28.703116 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:28.703853 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:28.705022 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:28.705428 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:28.705515 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:28.709273 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.216.130:43123: BlockingRecv error: recv got EOF from 127.24.216.130:43123 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:13:28.971877 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 1783
I20260929 02:13:28.982764 2243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:28.972809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34378 (local address 127.24.216.129:42551)
0929 02:13:28.973336 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:28.973343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:28.973367 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:13:28.973420 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:28.973426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:28.973654 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:28.973801 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:28.975088 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:28.976060 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:28.977114 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:28.977604 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:28.977694 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:28.980047 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:28.980055 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:28.980069 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:28.980096 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:28.982302 (+ 2206us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.216.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:13:28.982533 (+ 231us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34378: Request ticket server kudu/127.24.216.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":249,"threads_started":1}
W20260929 02:13:28.982877 2243 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34378: Request ticket server kudu/127.24.216.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:13:28.982885 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 1783: Not authorized: Client connection negotiation failed: client connection to 127.24.216.129:42551: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.216.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:13:28.983140 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 1783
I20260929 02:13:29.001053 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 1934
I20260929 02:13:29.012461 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:29.002076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.24.216.130:44365)
0929 02:13:29.002762 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:29.002769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:29.002799 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:13:29.002854 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:29.002860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:29.003077 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:29.003242 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:29.004396 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:29.005435 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:29.006696 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:29.007226 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:29.007329 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:29.009578 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:29.009586 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:29.009598 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:29.009627 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:29.011989 (+ 2362us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.216.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:13:29.012232 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44258: Request ticket server kudu/127.24.216.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":379,"threads_started":1}
W20260929 02:13:29.012595 2244 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44258: Request ticket server kudu/127.24.216.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:13:29.012596 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 1934: Not authorized: Client connection negotiation failed: client connection to 127.24.216.130:44365: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.216.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:13:29.012841 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 1934
I20260929 02:13:29.030339 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 2065
I20260929 02:13:29.041776 2245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:29.031226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.24.216.131:35541)
0929 02:13:29.031868 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:29.031875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:29.031901 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:13:29.031958 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:29.031965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:29.032189 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:29.032335 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:29.033503 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:29.034656 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:29.035743 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:29.036221 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:29.036350 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:29.038737 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:29.038746 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:29.038759 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:29.038793 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:29.041255 (+ 2462us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.216.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:13:29.041502 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44298: Request ticket server kudu/127.24.216.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":353,"threads_started":1}
W20260929 02:13:29.041877 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 2065: Not authorized: Client connection negotiation failed: client connection to 127.24.216.131:35541: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.24.216.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:13:29.041893 2245 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44298: Request ticket server kudu/127.24.216.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:13:29.042052 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 2065
I20260929 02:13:29.058053 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 1714
I20260929 02:13:29.059249 1776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:29.172940 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 1714
2026-09-29T02:13:29Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20982 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T02:13:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:13:29Z Disabled control of system clock
I20260929 02:13:29.254422 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:38227
--webserver_interface=127.24.216.190
--webserver_port=0
--builtin_ntp_servers=127.24.216.148:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:38227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 02:13:29.572227 2250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:29.572506 2250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:29.572578 2250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:29.579485 2250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:13:29.579581 2250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:29.579618 2250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:29.579670 2250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:13:29.579713 2250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:13:29.588421 2250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:38227
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:38227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:29.589589 2250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:29.591259 2250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:29.603849 2256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:29.603891 2258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:29.603946 2255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:29.604386 2250 server_base.cc:1034] running on GCE node
I20260929 02:13:29.605255 2250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:29.606583 2250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:29.607805 2250 hybrid_clock.cc:648] HybridClock initialized: now 1790648009607765 us; error 36 us; skew 500 ppm
I20260929 02:13:29.608145 2250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:13:29.610451 2250 webserver.cc:460] Webserver started at http://127.24.216.190:43423/ using document root <none> and password file <none>
I20260929 02:13:29.611119 2250 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:29.611248 2250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:29.611567 2250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:29.614427 2250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "6a5a40350d494205b60fa7d7b6941d07"
format_stamp: "Formatted at 2026-09-29 02:13:29 on dist-test-slave-9k0h"
server_key: "05819164060493888b73596274cbf5f1"
server_key_iv: "1979901a3baa0a4dc1e34dafe5708ea9"
server_key_version: "encryptionkey@0"
I20260929 02:13:29.615222 2250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "6a5a40350d494205b60fa7d7b6941d07"
format_stamp: "Formatted at 2026-09-29 02:13:29 on dist-test-slave-9k0h"
server_key: "05819164060493888b73596274cbf5f1"
server_key_iv: "1979901a3baa0a4dc1e34dafe5708ea9"
server_key_version: "encryptionkey@0"
I20260929 02:13:29.620052 2250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:13:29.623289 2264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:29.624455 2250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260929 02:13:29.624622 2250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "6a5a40350d494205b60fa7d7b6941d07"
format_stamp: "Formatted at 2026-09-29 02:13:29 on dist-test-slave-9k0h"
server_key: "05819164060493888b73596274cbf5f1"
server_key_iv: "1979901a3baa0a4dc1e34dafe5708ea9"
server_key_version: "encryptionkey@0"
I20260929 02:13:29.624817 2250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:29.640717 2250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:29.641450 2250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:29.641716 2250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:29.659350 2250 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:38227
I20260929 02:13:29.659394 2315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:38227 every 8 connection(s)
I20260929 02:13:29.660615 2250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:13:29.663691 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 2250
I20260929 02:13:29.663889 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:13:29.664301 2316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:29.664595 25442 external_mini_cluster.cc:1428] Setting key 2fabbb4e2c2eb9a2a15973485ee1dfdb
I20260929 02:13:29.676533 2316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07: Bootstrap starting.
I20260929 02:13:29.677268 2319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:29.666337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.24.216.190:38227)
0929 02:13:29.667252 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:29.667261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:29.667301 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:13:29.669138 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:29.669158 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:29.669593 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:29.669964 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:29.670543 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:29.671775 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:29.673053 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:29.673709 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:29.673920 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:29.675102 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:29.675109 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:29.675121 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:13:29.675616 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:29.675684 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:13:29.675703 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:29.675822 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:29.676036 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 02:13:29.676288 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":348,"threads_started":1}
I20260929 02:13:29.680109 2316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:29.681020 2316 log.cc:824] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07: Log is configured to *not* fsync() on all Append() calls
I20260929 02:13:29.683779 2316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07: No bootstrap required, opened a new log
I20260929 02:13:29.690577 2316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5a40350d494205b60fa7d7b6941d07" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 38227 } }
I20260929 02:13:29.690856 2316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:29.690922 2316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a5a40350d494205b60fa7d7b6941d07, State: Initialized, Role: FOLLOWER
I20260929 02:13:29.691350 2316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5a40350d494205b60fa7d7b6941d07" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 38227 } }
I20260929 02:13:29.691545 2316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:13:29.691633 2316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:13:29.691748 2316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:29.693642 2316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5a40350d494205b60fa7d7b6941d07" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 38227 } }
I20260929 02:13:29.694017 2316 leader_election.cc:302] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a5a40350d494205b60fa7d7b6941d07; no voters:
I20260929 02:13:29.694609 2316 leader_election.cc:288] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:13:29.694715 2321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:13:29.695392 2321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [term 1 LEADER]: Becoming Leader. State: Replica: 6a5a40350d494205b60fa7d7b6941d07, State: Running, Role: LEADER
I20260929 02:13:29.695904 2321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5a40350d494205b60fa7d7b6941d07" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 38227 } }
I20260929 02:13:29.696616 2316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:13:29.699101 2323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a5a40350d494205b60fa7d7b6941d07. Latest consensus state: current_term: 1 leader_uuid: "6a5a40350d494205b60fa7d7b6941d07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5a40350d494205b60fa7d7b6941d07" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 38227 } } }
I20260929 02:13:29.699404 2323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [sys.catalog]: This master's current role is: LEADER
I20260929 02:13:29.699872 2322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a5a40350d494205b60fa7d7b6941d07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5a40350d494205b60fa7d7b6941d07" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 38227 } } }
I20260929 02:13:29.700177 2322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07 [sys.catalog]: This master's current role is: LEADER
I20260929 02:13:29.700839 2328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:13:29.705706 2328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:13:29.714278 2328 catalog_manager.cc:1347] Generated new cluster ID: 3dbbdfcf9f79485397b19400652b5550
I20260929 02:13:29.714392 2328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:13:29.729626 2328 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:13:29.730974 2328 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:13:29.736702 2328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a5a40350d494205b60fa7d7b6941d07: Generated new TSK 0
I20260929 02:13:29.737427 2328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:13:29.744774 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:38227
--builtin_ntp_servers=127.24.216.148:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 02:13:30.098342 2340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:30.098593 2340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:30.098722 2340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:30.105841 2340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:30.105924 2340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:30.106076 2340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:13:30.114872 2340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:38227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:30.116122 2340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:30.117807 2340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:30.131057 2345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:30.131132 2346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:30.131101 2348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:30.131899 2340 server_base.cc:1034] running on GCE node
I20260929 02:13:30.132529 2340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:30.133462 2340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:30.134654 2340 hybrid_clock.cc:648] HybridClock initialized: now 1790648010134617 us; error 53 us; skew 500 ppm
I20260929 02:13:30.134974 2340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:13:30.137053 2340 webserver.cc:460] Webserver started at http://127.24.216.129:38527/ using document root <none> and password file <none>
I20260929 02:13:30.137712 2340 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:30.137837 2340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:30.138226 2340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:30.141088 2340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "6dd2372ca0694839b67e946cd4109bf1"
format_stamp: "Formatted at 2026-09-29 02:13:30 on dist-test-slave-9k0h"
server_key: "5e8236899ef68122f085725d478fd963"
server_key_iv: "502a4cbe11c5f01178f4303ec99c73b9"
server_key_version: "encryptionkey@0"
I20260929 02:13:30.141847 2340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "6dd2372ca0694839b67e946cd4109bf1"
format_stamp: "Formatted at 2026-09-29 02:13:30 on dist-test-slave-9k0h"
server_key: "5e8236899ef68122f085725d478fd963"
server_key_iv: "502a4cbe11c5f01178f4303ec99c73b9"
server_key_version: "encryptionkey@0"
I20260929 02:13:30.146792 2340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 02:13:30.149988 2354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:30.151374 2340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:13:30.151564 2340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "6dd2372ca0694839b67e946cd4109bf1"
format_stamp: "Formatted at 2026-09-29 02:13:30 on dist-test-slave-9k0h"
server_key: "5e8236899ef68122f085725d478fd963"
server_key_iv: "502a4cbe11c5f01178f4303ec99c73b9"
server_key_version: "encryptionkey@0"
I20260929 02:13:30.151782 2340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:30.169377 2340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:30.170305 2340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:30.170604 2340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:30.171474 2340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:30.172837 2340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:30.172953 2340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:30.173090 2340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:30.173178 2340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:30.205580 2340 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:37105
I20260929 02:13:30.205631 2466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:37105 every 8 connection(s)
I20260929 02:13:30.207073 2340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:13:30.210944 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 2340
I20260929 02:13:30.211109 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:13:30.211930 25442 external_mini_cluster.cc:1428] Setting key 74a81ca3b4dcab08daaf58776da5f349
I20260929 02:13:30.214834 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:38227
--builtin_ntp_servers=127.24.216.148:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 02:13:30.220952 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:30.209474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:48569 (local address 127.24.216.190:38227)
0929 02:13:30.209881 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:30.209887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:30.211465 (+ 1578us) server_negotiation.cc:407] Connection header received
0929 02:13:30.213598 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:30.213603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:30.213677 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:30.213791 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:30.215132 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:30.216374 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:30.217866 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:30.218571 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:30.218716 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:30.220037 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:30.220046 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:30.220054 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:13:30.220077 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:30.220125 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:13:30.220142 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:30.220236 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:30.220419 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 02:13:30.220520 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":170,"threads_started":1}
I20260929 02:13:30.221827 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:30.210200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:38227 (local address 127.24.216.129:48569)
0929 02:13:30.211269 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:30.211316 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:30.213316 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:30.214027 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:30.214037 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:30.214139 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:30.214922 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:30.214948 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:30.216531 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:30.216535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:30.217724 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:30.217738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:30.218765 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:30.218770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:30.218975 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:30.219731 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:13:30.219734 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:30.219770 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:13:30.219835 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:30.220237 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:30.220244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:30.220249 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:13:30.220378 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 02:13:30.220639 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":301,"threads_started":1}
I20260929 02:13:30.223315 2467 heartbeater.cc:347] Connected to a master server at 127.24.216.190:38227
I20260929 02:13:30.223605 2467 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:30.224294 2467 heartbeater.cc:511] Master 127.24.216.190:38227 requested a full tablet report, sending...
I20260929 02:13:30.226130 2281 ts_manager.cc:194] Registered new tserver with Master: 6dd2372ca0694839b67e946cd4109bf1 (127.24.216.129:37105)
I20260929 02:13:30.227663 2281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.216.129:48569
W20260929 02:13:30.537714 2471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:30.538024 2471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:30.538228 2471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:30.545467 2471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:30.545598 2471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:30.545786 2471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:13:30.554574 2471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:38227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:30.555929 2471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:30.557703 2471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:30.570549 2476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:30.570673 2479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:30.570690 2477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:30.571256 2471 server_base.cc:1034] running on GCE node
I20260929 02:13:30.571854 2471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:30.572788 2471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:30.574018 2471 hybrid_clock.cc:648] HybridClock initialized: now 1790648010573975 us; error 32 us; skew 500 ppm
I20260929 02:13:30.574369 2471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:13:30.576498 2471 webserver.cc:460] Webserver started at http://127.24.216.130:44617/ using document root <none> and password file <none>
I20260929 02:13:30.577189 2471 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:30.577319 2471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:30.577636 2471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:30.580399 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "a809f47c0ee44d768a6d14f71841f31d"
format_stamp: "Formatted at 2026-09-29 02:13:30 on dist-test-slave-9k0h"
server_key: "abc408beae073d85252c4cb20ad76002"
server_key_iv: "35b783c566997c4936e43e957ba17352"
server_key_version: "encryptionkey@0"
I20260929 02:13:30.581147 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "a809f47c0ee44d768a6d14f71841f31d"
format_stamp: "Formatted at 2026-09-29 02:13:30 on dist-test-slave-9k0h"
server_key: "abc408beae073d85252c4cb20ad76002"
server_key_iv: "35b783c566997c4936e43e957ba17352"
server_key_version: "encryptionkey@0"
I20260929 02:13:30.585968 2471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 02:13:30.589210 2485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:30.590481 2471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:13:30.590646 2471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "a809f47c0ee44d768a6d14f71841f31d"
format_stamp: "Formatted at 2026-09-29 02:13:30 on dist-test-slave-9k0h"
server_key: "abc408beae073d85252c4cb20ad76002"
server_key_iv: "35b783c566997c4936e43e957ba17352"
server_key_version: "encryptionkey@0"
I20260929 02:13:30.590859 2471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:30.615272 2471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:30.616063 2471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:30.616371 2471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:30.617261 2471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:30.618618 2471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:30.618736 2471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:30.618868 2471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:30.618963 2471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:30.651298 2471 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:45307
I20260929 02:13:30.651404 2597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:45307 every 8 connection(s)
I20260929 02:13:30.652565 2471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:13:30.659878 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 2471
I20260929 02:13:30.660045 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:13:30.660506 25442 external_mini_cluster.cc:1428] Setting key 81ee2294842d17af0f06669820fd4a28
I20260929 02:13:30.663507 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:38227
--builtin_ntp_servers=127.24.216.148:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 02:13:30.665609 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:30.655019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:39943 (local address 127.24.216.190:38227)
0929 02:13:30.655249 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:30.655253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:30.656626 (+ 1373us) server_negotiation.cc:407] Connection header received
0929 02:13:30.658287 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:30.658291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:30.658352 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:30.658439 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:30.659416 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:30.660534 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:30.662493 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:30.663023 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:30.663153 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:30.664615 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:30.664623 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:30.664631 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:13:30.664654 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:30.664695 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:13:30.664713 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:30.664829 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:30.665115 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 02:13:30.665241 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 02:13:30.666639 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:30.655562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:38227 (local address 127.24.216.130:39943)
0929 02:13:30.656485 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:30.656523 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:30.657997 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:30.658636 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:30.658643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:30.658730 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:30.659258 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:30.659273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:30.660738 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:30.660744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:30.662349 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:30.662364 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:30.663234 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:30.663240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:30.663507 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:30.664273 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:13:30.664277 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:30.664327 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:13:30.664413 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:30.664860 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:30.664871 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:30.664880 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:13:30.665076 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 02:13:30.665436 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":290,"threads_started":1}
I20260929 02:13:30.668000 2598 heartbeater.cc:347] Connected to a master server at 127.24.216.190:38227
I20260929 02:13:30.668274 2598 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:30.669018 2598 heartbeater.cc:511] Master 127.24.216.190:38227 requested a full tablet report, sending...
I20260929 02:13:30.670481 2281 ts_manager.cc:194] Registered new tserver with Master: a809f47c0ee44d768a6d14f71841f31d (127.24.216.130:45307)
I20260929 02:13:30.671657 2281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.216.130:39943
W20260929 02:13:30.982241 2601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:30.982509 2601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:30.982672 2601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:30.989910 2601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:30.990000 2601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:30.990139 2601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:13:30.999279 2601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:38227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:31.000552 2601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:31.002286 2601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:31.015027 2606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:31.015000 2609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:31.015090 2607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:31.019519 2601 server_base.cc:1034] running on GCE node
I20260929 02:13:31.020045 2601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:31.020861 2601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:31.022051 2601 hybrid_clock.cc:648] HybridClock initialized: now 1790648011022018 us; error 30 us; skew 500 ppm
I20260929 02:13:31.022392 2601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:13:31.031111 2601 webserver.cc:460] Webserver started at http://127.24.216.131:36823/ using document root <none> and password file <none>
I20260929 02:13:31.031769 2601 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:31.031867 2601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:31.032130 2601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:31.034879 2601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "8d7ccc9fe94d49ddaec6cd81b1f205c1"
format_stamp: "Formatted at 2026-09-29 02:13:31 on dist-test-slave-9k0h"
server_key: "33aba9ea22df31db8928edd2e77fcfa8"
server_key_iv: "ad719765bb0a7729b2bc887a5397bf9f"
server_key_version: "encryptionkey@0"
I20260929 02:13:31.035588 2601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "8d7ccc9fe94d49ddaec6cd81b1f205c1"
format_stamp: "Formatted at 2026-09-29 02:13:31 on dist-test-slave-9k0h"
server_key: "33aba9ea22df31db8928edd2e77fcfa8"
server_key_iv: "ad719765bb0a7729b2bc887a5397bf9f"
server_key_version: "encryptionkey@0"
I20260929 02:13:31.040477 2601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:13:31.043784 2615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:31.045028 2601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 02:13:31.045192 2601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "8d7ccc9fe94d49ddaec6cd81b1f205c1"
format_stamp: "Formatted at 2026-09-29 02:13:31 on dist-test-slave-9k0h"
server_key: "33aba9ea22df31db8928edd2e77fcfa8"
server_key_iv: "ad719765bb0a7729b2bc887a5397bf9f"
server_key_version: "encryptionkey@0"
I20260929 02:13:31.045404 2601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:31.060249 2601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:31.061026 2601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:31.061326 2601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:31.062237 2601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:31.063553 2601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:31.063634 2601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:31.063781 2601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:31.063872 2601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:31.095499 2601 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:34617
I20260929 02:13:31.095562 2727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:34617 every 8 connection(s)
I20260929 02:13:31.096789 2601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:13:31.098075 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 2601
I20260929 02:13:31.098241 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:13:31.098752 25442 external_mini_cluster.cc:1428] Setting key 198183c008f51bf1a302c7f8cd55e582
I20260929 02:13:31.108917 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:31.099767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:48807 (local address 127.24.216.190:38227)
0929 02:13:31.099973 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:31.099979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:31.101689 (+ 1710us) server_negotiation.cc:407] Connection header received
0929 02:13:31.103356 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:31.103359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:31.103418 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:31.103494 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:31.104390 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:31.105059 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:31.106581 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:31.107031 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:31.107120 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:31.108309 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:31.108312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:31.108315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:13:31.108329 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:31.108360 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:13:31.108370 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:31.108439 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:31.108615 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 02:13:31.108678 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:13:31.109768 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:31.100474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:38227 (local address 127.24.216.131:48807)
0929 02:13:31.101544 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:31.101586 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:31.103142 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:31.103671 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:31.103679 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:31.103753 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:31.104258 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:31.104273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:31.105174 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:31.105177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:31.106472 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:31.106482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:31.107194 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:31.107199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:31.107385 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:31.108058 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:13:31.108061 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:31.108096 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:13:31.108167 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:31.108451 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:31.108457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:31.108462 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:13:31.108584 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 02:13:31.108844 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":337,"threads_started":1}
I20260929 02:13:31.110953 2728 heartbeater.cc:347] Connected to a master server at 127.24.216.190:38227
I20260929 02:13:31.111207 2728 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:31.111894 2728 heartbeater.cc:511] Master 127.24.216.190:38227 requested a full tablet report, sending...
I20260929 02:13:31.113291 2281 ts_manager.cc:194] Registered new tserver with Master: 8d7ccc9fe94d49ddaec6cd81b1f205c1 (127.24.216.131:34617)
I20260929 02:13:31.113873 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:13:31.114315 2281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.216.131:48807
W20260929 02:13:31.119932 2469 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:31.118343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.24.216.190:38227)
0929 02:13:31.118500 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:31.118504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:31.118987 (+ 483us) server_negotiation.cc:407] Connection header received
0929 02:13:31.119162 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:31.119166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:31.119226 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:31.119319 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:31.119772 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37890: BlockingRecv error: recv got EOF from 127.0.0.1:37890 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:13:31.120426 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 2340
I20260929 02:13:31.127633 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:31.121428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.24.216.129:37105)
0929 02:13:31.121983 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:31.121989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:31.122012 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:13:31.122061 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:31.122066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:31.122293 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:31.122413 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:31.122688 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:31.123620 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:31.124762 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:31.125328 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:31.125424 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:31.126383 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:31.126390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:31.126401 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:13:31.126898 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:31.126937 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:13:31.126955 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:31.127068 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:31.127217 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 02:13:31.127357 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":291,"threads_started":1}
I20260929 02:13:31.128718 2462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:31.209604 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 2340
I20260929 02:13:31.229903 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 2471
I20260929 02:13:31.237298 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:31.230896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.24.216.130:45307)
0929 02:13:31.231393 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:31.231400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:31.231425 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:13:31.231481 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:31.231505 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:31.231716 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:31.231845 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:31.232183 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:31.233116 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:31.234301 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:31.234848 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:31.234945 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:31.235812 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:31.235820 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:31.235835 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:13:31.236592 (+ 757us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:31.236637 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:13:31.236669 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:31.236747 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:31.236859 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 02:13:31.236978 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":222,"threads_started":1}
I20260929 02:13:31.238497 2593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:31.318112 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 2471
I20260929 02:13:31.334950 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 2601
I20260929 02:13:31.342406 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:31.335827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.24.216.131:34617)
0929 02:13:31.336345 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:31.336365 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:31.336428 (+ 63us) server_negotiation.cc:407] Connection header received
0929 02:13:31.336509 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:31.336516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:31.336757 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:31.336888 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:31.337241 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:31.338302 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:31.339476 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:31.340009 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:31.340111 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:31.341005 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:31.341014 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:31.341026 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:13:31.341571 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:31.341615 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:13:31.341632 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:31.341780 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:31.341953 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 02:13:31.342048 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":219,"threads_started":1}
I20260929 02:13:31.343569 2723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:31.423694 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 2601
I20260929 02:13:31.440196 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 2250
I20260929 02:13:31.441290 2311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:31.561005 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 2250
2026-09-29T02:13:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2383 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T02:13:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:13:31Z Disabled control of system clock
I20260929 02:13:31.637374 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:41345
--webserver_interface=127.24.216.190
--webserver_port=0
--builtin_ntp_servers=127.24.216.148:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:41345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 02:13:31.958483 2747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:31.958813 2747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:31.958958 2747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:31.966042 2747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:13:31.966202 2747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:31.966274 2747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:31.966362 2747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:13:31.966454 2747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:13:31.975533 2747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:41345
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:41345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:31.976821 2747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:31.978600 2747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:31.989791 2752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:31.989791 2755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:31.989826 2753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:31.991096 2747 server_base.cc:1034] running on GCE node
I20260929 02:13:31.991869 2747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:31.993180 2747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:31.994397 2747 hybrid_clock.cc:648] HybridClock initialized: now 1790648011994356 us; error 55 us; skew 500 ppm
I20260929 02:13:31.994725 2747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:13:31.996902 2747 webserver.cc:460] Webserver started at http://127.24.216.190:40789/ using document root <none> and password file <none>
I20260929 02:13:31.997568 2747 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:31.997697 2747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:31.998021 2747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:32.000809 2747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "195636c51e6a4c1b9a5ee6d71da36326"
format_stamp: "Formatted at 2026-09-29 02:13:31 on dist-test-slave-9k0h"
server_key: "be4cd38fb1a8fe406a1a7b60932f3f20"
server_key_iv: "924792714523c5ca982bbabff7dcd07f"
server_key_version: "encryptionkey@0"
I20260929 02:13:32.001614 2747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "195636c51e6a4c1b9a5ee6d71da36326"
format_stamp: "Formatted at 2026-09-29 02:13:31 on dist-test-slave-9k0h"
server_key: "be4cd38fb1a8fe406a1a7b60932f3f20"
server_key_iv: "924792714523c5ca982bbabff7dcd07f"
server_key_version: "encryptionkey@0"
I20260929 02:13:32.006348 2747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 02:13:32.009538 2761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:32.010870 2747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:13:32.011039 2747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "195636c51e6a4c1b9a5ee6d71da36326"
format_stamp: "Formatted at 2026-09-29 02:13:31 on dist-test-slave-9k0h"
server_key: "be4cd38fb1a8fe406a1a7b60932f3f20"
server_key_iv: "924792714523c5ca982bbabff7dcd07f"
server_key_version: "encryptionkey@0"
I20260929 02:13:32.011246 2747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:32.016703 2747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:32.017401 2747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:32.017674 2747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:32.035183 2747 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:41345
I20260929 02:13:32.035218 2812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:41345 every 8 connection(s)
I20260929 02:13:32.036516 2747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:13:32.040158 2813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:32.046816 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 2747
I20260929 02:13:32.047037 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:13:32.047622 25442 external_mini_cluster.cc:1428] Setting key 9466f9a59b82d46a4030514ab905150a
I20260929 02:13:32.050187 2813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326: Bootstrap starting.
I20260929 02:13:32.053318 2813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:32.054581 2813 log.cc:824] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326: Log is configured to *not* fsync() on all Append() calls
I20260929 02:13:32.055677 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:32.049513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.24.216.190:41345)
0929 02:13:32.050526 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:32.050537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:32.050575 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:13:32.052161 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:32.052182 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:32.052555 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:32.052861 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:32.053947 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:32.053952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:32.053964 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:13:32.053976 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:32.054026 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:13:32.054044 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:32.054233 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:32.054340 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 02:13:32.054607 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":331,"threads_started":1}
I20260929 02:13:32.057463 2813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326: No bootstrap required, opened a new log
I20260929 02:13:32.064399 2813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195636c51e6a4c1b9a5ee6d71da36326" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41345 } }
I20260929 02:13:32.064724 2813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:32.064819 2813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195636c51e6a4c1b9a5ee6d71da36326, State: Initialized, Role: FOLLOWER
I20260929 02:13:32.065320 2813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195636c51e6a4c1b9a5ee6d71da36326" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41345 } }
I20260929 02:13:32.065534 2813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:13:32.065662 2813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:13:32.065817 2813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:32.067837 2813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195636c51e6a4c1b9a5ee6d71da36326" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41345 } }
I20260929 02:13:32.068320 2813 leader_election.cc:302] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 195636c51e6a4c1b9a5ee6d71da36326; no voters:
I20260929 02:13:32.068867 2813 leader_election.cc:288] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:13:32.068990 2818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:13:32.069667 2818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [term 1 LEADER]: Becoming Leader. State: Replica: 195636c51e6a4c1b9a5ee6d71da36326, State: Running, Role: LEADER
I20260929 02:13:32.070328 2818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195636c51e6a4c1b9a5ee6d71da36326" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41345 } }
I20260929 02:13:32.071041 2813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:13:32.072999 2819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "195636c51e6a4c1b9a5ee6d71da36326" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195636c51e6a4c1b9a5ee6d71da36326" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41345 } } }
I20260929 02:13:32.073068 2820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 195636c51e6a4c1b9a5ee6d71da36326. Latest consensus state: current_term: 1 leader_uuid: "195636c51e6a4c1b9a5ee6d71da36326" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195636c51e6a4c1b9a5ee6d71da36326" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41345 } } }
I20260929 02:13:32.073462 2819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [sys.catalog]: This master's current role is: LEADER
I20260929 02:13:32.073488 2820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326 [sys.catalog]: This master's current role is: LEADER
I20260929 02:13:32.074748 2827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:13:32.080773 2827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:13:32.090338 2827 catalog_manager.cc:1347] Generated new cluster ID: a412f123846e4abdad92ea2edf5896b8
I20260929 02:13:32.090441 2827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:13:32.099790 2827 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:13:32.100211 2827 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:13:32.112560 2827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 195636c51e6a4c1b9a5ee6d71da36326: Generated new TSK 0
I20260929 02:13:32.113466 2827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:13:32.127420 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41345
--builtin_ntp_servers=127.24.216.148:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 02:13:32.449991 2837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:32.450294 2837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:32.450469 2837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:32.457275 2837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:32.457366 2837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:32.457506 2837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:13:32.466117 2837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:32.467408 2837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:32.469168 2837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:32.481604 2843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:32.481673 2842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:32.482062 2845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:32.482662 2837 server_base.cc:1034] running on GCE node
I20260929 02:13:32.483243 2837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:32.484098 2837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:32.485311 2837 hybrid_clock.cc:648] HybridClock initialized: now 1790648012485267 us; error 57 us; skew 500 ppm
I20260929 02:13:32.485600 2837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:13:32.487623 2837 webserver.cc:460] Webserver started at http://127.24.216.129:33757/ using document root <none> and password file <none>
I20260929 02:13:32.488284 2837 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:32.488409 2837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:32.488729 2837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:32.491402 2837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "3a714644f7c44cfab844ff19b2f031c8"
format_stamp: "Formatted at 2026-09-29 02:13:32 on dist-test-slave-9k0h"
server_key: "a5f0eb27365447652288fc768cc34094"
server_key_iv: "c8c17868177e854d16d79862d4207bfc"
server_key_version: "encryptionkey@0"
I20260929 02:13:32.492163 2837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "3a714644f7c44cfab844ff19b2f031c8"
format_stamp: "Formatted at 2026-09-29 02:13:32 on dist-test-slave-9k0h"
server_key: "a5f0eb27365447652288fc768cc34094"
server_key_iv: "c8c17868177e854d16d79862d4207bfc"
server_key_version: "encryptionkey@0"
I20260929 02:13:32.496914 2837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:13:32.500083 2851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:32.501245 2837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:13:32.501413 2837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "3a714644f7c44cfab844ff19b2f031c8"
format_stamp: "Formatted at 2026-09-29 02:13:32 on dist-test-slave-9k0h"
server_key: "a5f0eb27365447652288fc768cc34094"
server_key_iv: "c8c17868177e854d16d79862d4207bfc"
server_key_version: "encryptionkey@0"
I20260929 02:13:32.501611 2837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:32.507017 2837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:32.507717 2837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:32.507973 2837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:32.508867 2837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:32.510207 2837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:32.510298 2837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:32.510375 2837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:32.510427 2837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:32.541590 2837 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:34569
I20260929 02:13:32.541635 2963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:34569 every 8 connection(s)
I20260929 02:13:32.542923 2837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:13:32.550324 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 2837
I20260929 02:13:32.550547 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:13:32.550729 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:32.545442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:34471 (local address 127.24.216.190:41345)
0929 02:13:32.545610 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:32.545614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:32.547020 (+ 1406us) server_negotiation.cc:407] Connection header received
0929 02:13:32.548586 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:32.548591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:32.548670 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:32.548782 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:32.549986 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:32.549990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:32.549993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:13:32.550007 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:32.550036 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:13:32.550047 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:32.550188 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:32.550391 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 02:13:32.550468 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:13:32.551326 25442 external_mini_cluster.cc:1428] Setting key 8fdac10d1c7e6d4f08a2d65ca6e96abe
I20260929 02:13:32.551869 2966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:32.545958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:41345 (local address 127.24.216.129:34471)
0929 02:13:32.546870 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:32.546916 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:32.548334 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:32.549018 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:32.549025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:32.549107 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:32.549679 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:13:32.549681 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:32.549716 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:13:32.549792 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:32.550170 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:32.550184 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:32.550192 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 02:13:32.550468 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 02:13:32.550710 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":318,"threads_started":1}
I20260929 02:13:32.553399 2964 heartbeater.cc:347] Connected to a master server at 127.24.216.190:41345
I20260929 02:13:32.553683 2964 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:32.554387 2964 heartbeater.cc:511] Master 127.24.216.190:41345 requested a full tablet report, sending...
I20260929 02:13:32.554387 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41345
--builtin_ntp_servers=127.24.216.148:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 02:13:32.556537 2778 ts_manager.cc:194] Registered new tserver with Master: 3a714644f7c44cfab844ff19b2f031c8 (127.24.216.129:34569)
W20260929 02:13:32.872233 2967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:32.872514 2967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:32.872696 2967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:32.880079 2967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:32.880173 2967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:32.880373 2967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:13:32.889392 2967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:32.890743 2967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:32.892474 2967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:32.905453 2972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:32.905491 2975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:32.905539 2973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:32.911578 2967 server_base.cc:1034] running on GCE node
I20260929 02:13:32.912201 2967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:32.913111 2967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:32.914289 2967 hybrid_clock.cc:648] HybridClock initialized: now 1790648012914251 us; error 31 us; skew 500 ppm
I20260929 02:13:32.914594 2967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:13:32.916966 2967 webserver.cc:460] Webserver started at http://127.24.216.130:44483/ using document root <none> and password file <none>
I20260929 02:13:32.917608 2967 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:32.917723 2967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:32.918042 2967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:32.920820 2967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "2a37a218761c4534a5d2828be8ba425d"
format_stamp: "Formatted at 2026-09-29 02:13:32 on dist-test-slave-9k0h"
server_key: "27dc5fbdf885a890e241784518ad063d"
server_key_iv: "07ebb12ebad0ceb6cdd5162f307ccc39"
server_key_version: "encryptionkey@0"
I20260929 02:13:32.921582 2967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "2a37a218761c4534a5d2828be8ba425d"
format_stamp: "Formatted at 2026-09-29 02:13:32 on dist-test-slave-9k0h"
server_key: "27dc5fbdf885a890e241784518ad063d"
server_key_iv: "07ebb12ebad0ceb6cdd5162f307ccc39"
server_key_version: "encryptionkey@0"
I20260929 02:13:32.926811 2967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:13:32.929976 2981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:32.931255 2967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:13:32.931442 2967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "2a37a218761c4534a5d2828be8ba425d"
format_stamp: "Formatted at 2026-09-29 02:13:32 on dist-test-slave-9k0h"
server_key: "27dc5fbdf885a890e241784518ad063d"
server_key_iv: "07ebb12ebad0ceb6cdd5162f307ccc39"
server_key_version: "encryptionkey@0"
I20260929 02:13:32.931667 2967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:32.937016 2967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:32.937760 2967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:32.938023 2967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:32.938921 2967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:32.940348 2967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:32.940469 2967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:32.940605 2967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:32.940703 2967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:32.972726 2967 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:46267
I20260929 02:13:32.972756 3093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:46267 every 8 connection(s)
I20260929 02:13:32.974045 2967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:13:32.978315 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 2967
I20260929 02:13:32.978502 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:13:32.979050 25442 external_mini_cluster.cc:1428] Setting key 0df67597d2af82bac86b526f32872c17
I20260929 02:13:32.981971 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41345
--builtin_ntp_servers=127.24.216.148:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 02:13:32.982844 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:32.976615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:59261 (local address 127.24.216.190:41345)
0929 02:13:32.976776 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:32.976780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:32.978361 (+ 1581us) server_negotiation.cc:407] Connection header received
0929 02:13:32.980368 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:32.980374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:32.980432 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:32.980550 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:32.981998 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:32.982002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:32.982006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:13:32.982020 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:32.982077 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:13:32.982093 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:32.982278 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:32.982510 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 02:13:32.982575 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:13:32.983795 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:32.977192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:41345 (local address 127.24.216.130:59261)
0929 02:13:32.978182 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:32.978227 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:32.980112 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:32.980796 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:32.980803 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:32.980930 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:32.981650 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:13:32.981656 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:32.981711 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:13:32.981823 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:32.982274 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:32.982286 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:32.982296 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:13:32.982577 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 02:13:32.982861 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":248,"threads_started":1}
I20260929 02:13:32.985116 3094 heartbeater.cc:347] Connected to a master server at 127.24.216.190:41345
I20260929 02:13:32.985437 3094 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:32.986096 3094 heartbeater.cc:511] Master 127.24.216.190:41345 requested a full tablet report, sending...
I20260929 02:13:32.987622 2778 ts_manager.cc:194] Registered new tserver with Master: 2a37a218761c4534a5d2828be8ba425d (127.24.216.130:46267)
W20260929 02:13:33.299312 3097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:33.299592 3097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:33.299748 3097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:33.306991 3097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:33.307075 3097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:33.307271 3097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:13:33.316116 3097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:33.317309 3097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:33.319054 3097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:33.331434 3102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:33.331486 3103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:33.331514 3105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:33.332621 3097 server_base.cc:1034] running on GCE node
I20260929 02:13:33.333125 3097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:33.333927 3097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:33.335168 3097 hybrid_clock.cc:648] HybridClock initialized: now 1790648013335093 us; error 59 us; skew 500 ppm
I20260929 02:13:33.335544 3097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:13:33.337598 3097 webserver.cc:460] Webserver started at http://127.24.216.131:42149/ using document root <none> and password file <none>
I20260929 02:13:33.338307 3097 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:33.338425 3097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:33.338716 3097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:33.341436 3097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "2f13e9e8505f42409f1eef5338d0b5c8"
format_stamp: "Formatted at 2026-09-29 02:13:33 on dist-test-slave-9k0h"
server_key: "73c3e76b23f01189770a528980fc3228"
server_key_iv: "0eb379234c947a0d1f152c4fac669298"
server_key_version: "encryptionkey@0"
I20260929 02:13:33.342272 3097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "2f13e9e8505f42409f1eef5338d0b5c8"
format_stamp: "Formatted at 2026-09-29 02:13:33 on dist-test-slave-9k0h"
server_key: "73c3e76b23f01189770a528980fc3228"
server_key_iv: "0eb379234c947a0d1f152c4fac669298"
server_key_version: "encryptionkey@0"
I20260929 02:13:33.347051 3097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:13:33.350139 3111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:33.351336 3097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 02:13:33.351492 3097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "2f13e9e8505f42409f1eef5338d0b5c8"
format_stamp: "Formatted at 2026-09-29 02:13:33 on dist-test-slave-9k0h"
server_key: "73c3e76b23f01189770a528980fc3228"
server_key_iv: "0eb379234c947a0d1f152c4fac669298"
server_key_version: "encryptionkey@0"
I20260929 02:13:33.351733 3097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:33.357007 3097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:33.357704 3097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:33.357959 3097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:33.358824 3097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:33.360100 3097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:33.360222 3097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:33.360361 3097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:33.360450 3097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:33.392231 3097 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:38223
I20260929 02:13:33.392277 3223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:38223 every 8 connection(s)
I20260929 02:13:33.393504 3097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:13:33.395617 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 3097
I20260929 02:13:33.395795 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:13:33.396322 25442 external_mini_cluster.cc:1428] Setting key 59e9cd4109da3ba35d2078a3aad61802
I20260929 02:13:33.401547 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:33.396253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:53607 (local address 127.24.216.190:41345)
0929 02:13:33.396454 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:33.396460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:33.398081 (+ 1621us) server_negotiation.cc:407] Connection header received
0929 02:13:33.399768 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:33.399772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:33.399820 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:33.399897 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:33.401020 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:33.401024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:33.401027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:13:33.401040 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:33.401067 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:13:33.401077 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:33.401159 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:33.401321 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 02:13:33.401371 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:13:33.402536 3226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:33.396943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:41345 (local address 127.24.216.131:53607)
0929 02:13:33.397929 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:33.397970 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:33.399548 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:33.400087 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:33.400095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:33.400191 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:33.400769 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:13:33.400772 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:33.400805 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:13:33.400883 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:33.401163 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:33.401169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:33.401175 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 02:13:33.401364 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 02:13:33.401536 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":321,"threads_started":1}
I20260929 02:13:33.403698 3224 heartbeater.cc:347] Connected to a master server at 127.24.216.190:41345
I20260929 02:13:33.403956 3224 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:33.404518 3224 heartbeater.cc:511] Master 127.24.216.190:41345 requested a full tablet report, sending...
I20260929 02:13:33.405781 2778 ts_manager.cc:194] Registered new tserver with Master: 2f13e9e8505f42409f1eef5338d0b5c8 (127.24.216.131:38223)
I20260929 02:13:33.411168 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 02:13:33.416976 2816 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:13:33.415516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.24.216.190:41345)
0929 02:13:33.415650 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:33.415654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:33.416193 (+ 539us) server_negotiation.cc:407] Connection header received
0929 02:13:33.416259 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:33.416265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:33.416326 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:33.416421 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:33.416774 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47944: BlockingRecv error: recv got EOF from 127.0.0.1:47944 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:13:33.417320 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 2837
I20260929 02:13:33.420519 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:33.418232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.24.216.129:34569)
0929 02:13:33.418718 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:33.418724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:33.418747 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:13:33.418797 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:33.418802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:33.418966 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:33.419073 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:33.419896 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:33.419902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:33.419911 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:13:33.419927 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:33.419962 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:13:33.419975 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:33.420070 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:33.420145 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 02:13:33.420245 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":232,"threads_started":1}
I20260929 02:13:33.421772 2959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:33.501250 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 2837
I20260929 02:13:33.517210 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 2967
I20260929 02:13:33.520637 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:33.518013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.24.216.130:46267)
0929 02:13:33.518525 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:33.518532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:33.518557 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:13:33.518615 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:33.518621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:33.518826 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:33.518984 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:33.519969 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:33.519975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:33.519985 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:13:33.520001 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:33.520037 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:13:33.520050 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:33.520187 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:33.520288 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 02:13:33.520393 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":280,"threads_started":1}
I20260929 02:13:33.521785 3089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:33.599257 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 2967
I20260929 02:13:33.617074 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 3097
I20260929 02:13:33.620819 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:33.617870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.24.216.131:38223)
0929 02:13:33.618481 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:33.618494 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:33.618522 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:13:33.618582 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:33.618588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:33.618804 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:33.618968 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:33.620078 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:33.620083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:33.620095 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:13:33.620106 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:33.620145 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:13:33.620159 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:33.620356 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:33.620425 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 02:13:33.620528 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":326,"threads_started":1}
I20260929 02:13:33.622066 3219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:33.701488 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 3097
I20260929 02:13:33.717959 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 2747
I20260929 02:13:33.719177 2808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:33.817566 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 2747
2026-09-29T02:13:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2255 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:13:33 dist-test-slave-9k0h krb5kdc[3242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:13:33 dist-test-slave-9k0h krb5kdc[3242](info): set up 2 sockets
Sep 29 02:13:33 dist-test-slave-9k0h krb5kdc[3242](info): commencing operation
krb5kdc: starting...
W20260929 02:13:34.919078 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:13:35 dist-test-slave-9k0h krb5kdc[3242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:13:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:13:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:13:35.137782 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:42755
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:42755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:13:35.458940 3255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:35.459194 3255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:35.459280 3255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:35.466279 3255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:13:35.466372 3255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:35.466418 3255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:35.466478 3255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:13:35.466521 3255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:13:35.475674 3255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:42755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:42755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:35.476799 3255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:35.478003 3255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:35.489506 3260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:35.489579 3263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:35.489601 3261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:35.490172 3255 server_base.cc:1034] running on GCE node
I20260929 02:13:35.490895 3255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:35.492240 3255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:35.493472 3255 hybrid_clock.cc:648] HybridClock initialized: now 1790648015493432 us; error 43 us; skew 500 ppm
I20260929 02:13:35.493804 3255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:35 dist-test-slave-9k0h krb5kdc[3242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648015, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:35.496773 3255 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:13:35.498251 3255 webserver.cc:460] Webserver started at http://127.24.216.190:33087/ using document root <none> and password file <none>
I20260929 02:13:35.498921 3255 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:35.499068 3255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:35.499372 3255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:35.502297 3255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "184cd401e7cb42a0a6210e8867ef4b3d"
format_stamp: "Formatted at 2026-09-29 02:13:35 on dist-test-slave-9k0h"
server_key: "a1f8211373f20b675e1a0c74c78d4fcf"
server_key_iv: "7d3da2ee2c75746ee2cf63d820f70e10"
server_key_version: "encryptionkey@0"
I20260929 02:13:35.503057 3255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "184cd401e7cb42a0a6210e8867ef4b3d"
format_stamp: "Formatted at 2026-09-29 02:13:35 on dist-test-slave-9k0h"
server_key: "a1f8211373f20b675e1a0c74c78d4fcf"
server_key_iv: "7d3da2ee2c75746ee2cf63d820f70e10"
server_key_version: "encryptionkey@0"
I20260929 02:13:35.507809 3255 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:13:35.511073 3270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:35.512334 3255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 02:13:35.512521 3255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "184cd401e7cb42a0a6210e8867ef4b3d"
format_stamp: "Formatted at 2026-09-29 02:13:35 on dist-test-slave-9k0h"
server_key: "a1f8211373f20b675e1a0c74c78d4fcf"
server_key_iv: "7d3da2ee2c75746ee2cf63d820f70e10"
server_key_version: "encryptionkey@0"
I20260929 02:13:35.512723 3255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:35.525185 3255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:35.528934 3255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:35.529246 3255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:35.547042 3255 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:42755
I20260929 02:13:35.547070 3321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:42755 every 8 connection(s)
I20260929 02:13:35.548346 3255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:13:35.549636 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 3255
I20260929 02:13:35.549850 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:13:35.550678 25442 external_mini_cluster.cc:1428] Setting key 8bd20b3959d8214d7430265eeda765e5
I20260929 02:13:35.552881 3322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:13:35 dist-test-slave-9k0h krb5kdc[3242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:13:35.565615 3322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d: Bootstrap starting.
I20260929 02:13:35.568477 3322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:35.569433 3322 log.cc:824] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:13:35.572283 3322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d: No bootstrap required, opened a new log
I20260929 02:13:35.572991 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:35.552662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58160 (local address 127.24.216.190:42755)
0929 02:13:35.553471 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:35.553482 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:35.553521 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:13:35.555067 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:35.555086 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:35.555448 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:35.555838 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:35.557546 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:35.558956 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:35.560667 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:35.561272 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:35.561399 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:35.565818 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:35.565826 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:35.565839 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:35.565871 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:35.568883 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:35.569578 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:35.569584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:35.569590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:35.569771 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:35.570185 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:35.570190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:35.570194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:35.570854 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:35.571181 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:35.571621 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 02:13:35.572014 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":397,"threads_started":1}
I20260929 02:13:35.579491 3322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184cd401e7cb42a0a6210e8867ef4b3d" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42755 } }
I20260929 02:13:35.579830 3322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:35.579941 3322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184cd401e7cb42a0a6210e8867ef4b3d, State: Initialized, Role: FOLLOWER
I20260929 02:13:35.580446 3322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184cd401e7cb42a0a6210e8867ef4b3d" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42755 } }
I20260929 02:13:35.580718 3322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:13:35.580878 3322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:13:35.581043 3322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:35.583060 3322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184cd401e7cb42a0a6210e8867ef4b3d" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42755 } }
I20260929 02:13:35.583550 3322 leader_election.cc:302] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 184cd401e7cb42a0a6210e8867ef4b3d; no voters:
I20260929 02:13:35.584231 3322 leader_election.cc:288] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:13:35.584342 3327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:13:35.584987 3327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [term 1 LEADER]: Becoming Leader. State: Replica: 184cd401e7cb42a0a6210e8867ef4b3d, State: Running, Role: LEADER
I20260929 02:13:35.585474 3327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184cd401e7cb42a0a6210e8867ef4b3d" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42755 } }
I20260929 02:13:35.586063 3322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:13:35.588052 3329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 184cd401e7cb42a0a6210e8867ef4b3d. Latest consensus state: current_term: 1 leader_uuid: "184cd401e7cb42a0a6210e8867ef4b3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184cd401e7cb42a0a6210e8867ef4b3d" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42755 } } }
I20260929 02:13:35.588366 3329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [sys.catalog]: This master's current role is: LEADER
I20260929 02:13:35.593166 3334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:13:35.592964 3328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "184cd401e7cb42a0a6210e8867ef4b3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184cd401e7cb42a0a6210e8867ef4b3d" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 42755 } } }
I20260929 02:13:35.593349 3328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d [sys.catalog]: This master's current role is: LEADER
I20260929 02:13:35.598286 3334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:13:35.606659 3334 catalog_manager.cc:1347] Generated new cluster ID: 1c9044b5a4284837a6f5548fb62eea34
I20260929 02:13:35.606767 3334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:13:35.619481 3334 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:13:35.620649 3334 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:13:35.636983 3334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d: Generated new TSK 0
I20260929 02:13:35.637763 3334 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:13:35.697777 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:42755
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:13:36.035310 3348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:36.035566 3348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:36.035727 3348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:36.042721 3348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:36.042814 3348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:36.043030 3348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:13:36.052726 3348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:42755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:36.053969 3348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:36.055238 3348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:36.067684 3354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:36.067734 3353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:36.067768 3356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:36.068660 3348 server_base.cc:1034] running on GCE node
I20260929 02:13:36.069211 3348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:36.070106 3348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:36.071381 3348 hybrid_clock.cc:648] HybridClock initialized: now 1790648016071294 us; error 49 us; skew 500 ppm
I20260929 02:13:36.071689 3348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:36 dist-test-slave-9k0h krb5kdc[3242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648016, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:36.081300 3348 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:13:36.082790 3348 webserver.cc:460] Webserver started at http://127.24.216.129:45993/ using document root <none> and password file <none>
I20260929 02:13:36.083454 3348 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:36.083593 3348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:36.083875 3348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:36.086771 3348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "59b5a7a3f19f4416ab5d0dec36b58a01"
format_stamp: "Formatted at 2026-09-29 02:13:36 on dist-test-slave-9k0h"
server_key: "4af07563ef7aa5c74ef24c35920539d8"
server_key_iv: "363c269d4e5aef396d671cb2c201cf56"
server_key_version: "encryptionkey@0"
I20260929 02:13:36.087524 3348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "59b5a7a3f19f4416ab5d0dec36b58a01"
format_stamp: "Formatted at 2026-09-29 02:13:36 on dist-test-slave-9k0h"
server_key: "4af07563ef7aa5c74ef24c35920539d8"
server_key_iv: "363c269d4e5aef396d671cb2c201cf56"
server_key_version: "encryptionkey@0"
I20260929 02:13:36.092288 3348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:13:36.095660 3363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:36.096922 3348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:13:36.097184 3348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "59b5a7a3f19f4416ab5d0dec36b58a01"
format_stamp: "Formatted at 2026-09-29 02:13:36 on dist-test-slave-9k0h"
server_key: "4af07563ef7aa5c74ef24c35920539d8"
server_key_iv: "363c269d4e5aef396d671cb2c201cf56"
server_key_version: "encryptionkey@0"
I20260929 02:13:36.097365 3348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:36.111743 3348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:36.115870 3348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:36.116227 3348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:36.117100 3348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:36.118425 3348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:36.118494 3348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:36.118571 3348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:36.118618 3348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:36.150688 3348 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:42749
I20260929 02:13:36.150729 3475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:42749 every 8 connection(s)
I20260929 02:13:36.152045 3348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:13:36.153075 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 3348
I20260929 02:13:36.153282 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:13:36.153825 25442 external_mini_cluster.cc:1428] Setting key 60da5f49c5508fed64d8661fb82f13f2
Sep 29 02:13:36 dist-test-slave-9k0h krb5kdc[3242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648016, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:13:36.175788 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:36.154660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:41301 (local address 127.24.216.190:42755)
0929 02:13:36.155147 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:36.155151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:36.156259 (+ 1108us) server_negotiation.cc:407] Connection header received
0929 02:13:36.159106 (+ 2847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:36.159110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:36.159166 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:36.159255 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:36.161241 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:36.161990 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:36.164509 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:36.164899 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:36.164996 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:36.168882 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:36.168891 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:36.168898 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:36.168944 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:36.172566 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:36.173269 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:36.173278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:36.173284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:36.173450 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:36.173842 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:36.173847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:36.173852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:36.174253 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:36.174403 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:36.175054 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 02:13:36.175296 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":271,"threads_started":1}
I20260929 02:13:36.176394 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:36.155248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:42755 (local address 127.24.216.129:41301)
0929 02:13:36.156092 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:36.156145 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:36.158812 (+ 2667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:36.159526 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:36.159541 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:36.160286 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:36.161075 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:36.161102 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:36.162179 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:36.162187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:36.164359 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:36.164377 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:36.165146 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:36.165155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:36.165487 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:36.166307 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:36.166329 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:36.168653 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:36.172713 (+ 4060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:36.172720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:36.172733 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:36.173136 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:36.173557 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:36.173560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:36.173562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:36.173730 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:36.174375 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:36.174382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:36.174685 (+ 303us) client_negotiation.cc:769] Sending connection context
0929 02:13:36.174909 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 02:13:36.175281 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":333,"threads_started":1}
I20260929 02:13:36.178069 3476 heartbeater.cc:347] Connected to a master server at 127.24.216.190:42755
I20260929 02:13:36.178395 3476 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:36.179114 3476 heartbeater.cc:511] Master 127.24.216.190:42755 requested a full tablet report, sending...
I20260929 02:13:36.181139 3287 ts_manager.cc:194] Registered new tserver with Master: 59b5a7a3f19f4416ab5d0dec36b58a01 (127.24.216.129:42749)
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
I20260929 02:13:36.182776 3287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:41301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:13:36.211472 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:42755
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:13:36.532563 3482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:36.532824 3482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:36.532980 3482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:36.540020 3482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:36.540117 3482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:36.540315 3482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:13:36.549782 3482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:42755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:36.551030 3482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:36.552404 3482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:36.565544 3490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:36.565613 3488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:36.565675 3487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:36.566577 3482 server_base.cc:1034] running on GCE node
I20260929 02:13:36.567245 3482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:36.568104 3482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:36.569321 3482 hybrid_clock.cc:648] HybridClock initialized: now 1790648016569282 us; error 52 us; skew 500 ppm
I20260929 02:13:36.569619 3482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:36 dist-test-slave-9k0h krb5kdc[3242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648016, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:36.572678 3482 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:13:36.573987 3482 webserver.cc:460] Webserver started at http://127.24.216.130:34285/ using document root <none> and password file <none>
I20260929 02:13:36.574702 3482 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:36.574852 3482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:36.575187 3482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:36.578009 3482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "6098e0b8769d4a91aec244bcab43ed81"
format_stamp: "Formatted at 2026-09-29 02:13:36 on dist-test-slave-9k0h"
server_key: "8238afd5a59f2afb88a27f1a90430c1a"
server_key_iv: "9b6bbd88fc0d1af7eec69782b19e16a2"
server_key_version: "encryptionkey@0"
I20260929 02:13:36.578846 3482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "6098e0b8769d4a91aec244bcab43ed81"
format_stamp: "Formatted at 2026-09-29 02:13:36 on dist-test-slave-9k0h"
server_key: "8238afd5a59f2afb88a27f1a90430c1a"
server_key_iv: "9b6bbd88fc0d1af7eec69782b19e16a2"
server_key_version: "encryptionkey@0"
I20260929 02:13:36.583840 3482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:13:36.587240 3497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:36.588444 3482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:13:36.588615 3482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "6098e0b8769d4a91aec244bcab43ed81"
format_stamp: "Formatted at 2026-09-29 02:13:36 on dist-test-slave-9k0h"
server_key: "8238afd5a59f2afb88a27f1a90430c1a"
server_key_iv: "9b6bbd88fc0d1af7eec69782b19e16a2"
server_key_version: "encryptionkey@0"
I20260929 02:13:36.588809 3482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:36.603529 3482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:36.607571 3482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:36.607848 3482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:36.608776 3482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:36.610014 3482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:36.610188 3482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:36.610347 3482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:36.610421 3482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:36.642686 3482 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:36965
I20260929 02:13:36.642722 3609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:36965 every 8 connection(s)
I20260929 02:13:36.643977 3482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:13:36.652694 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 3482
I20260929 02:13:36.652894 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:13:36.653440 25442 external_mini_cluster.cc:1428] Setting key a81285ff8fb500d1a2885530ba692630
Sep 29 02:13:36 dist-test-slave-9k0h krb5kdc[3242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648016, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:13:36.665342 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:36.646375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:46693 (local address 127.24.216.190:42755)
0929 02:13:36.646540 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:36.646544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:36.647694 (+ 1150us) server_negotiation.cc:407] Connection header received
0929 02:13:36.649560 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:36.649566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:36.649653 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:36.649760 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:36.651199 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:36.651918 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:36.654205 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:36.654726 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:36.654854 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:36.658903 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:36.658912 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:36.658919 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:36.658963 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:36.662388 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:36.663209 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:36.663217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:36.663223 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:36.663373 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:36.663762 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:36.663769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:36.663775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:36.664129 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:36.664275 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:36.664904 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 02:13:36.665089 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:13:36.666749 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:36.646875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:42755 (local address 127.24.216.130:46693)
0929 02:13:36.647545 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:36.647590 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:36.649304 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:36.649984 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:36.649991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:36.650514 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:36.651045 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:36.651060 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:36.652059 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:36.652069 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:36.654026 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:36.654036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:36.654957 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:36.654964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:36.655173 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:36.655942 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:36.655988 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:36.658675 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:36.662513 (+ 3838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:36.662519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:36.662532 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:36.663062 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:36.663477 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:36.663480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:36.663482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:36.663649 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:36.664244 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:36.664249 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:36.664549 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 02:13:36.664755 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 02:13:36.665301 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":263,"threads_started":1}
I20260929 02:13:36.668066 3610 heartbeater.cc:347] Connected to a master server at 127.24.216.190:42755
I20260929 02:13:36.668339 3610 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:36.669023 3610 heartbeater.cc:511] Master 127.24.216.190:42755 requested a full tablet report, sending...
I20260929 02:13:36.670449 3287 ts_manager.cc:194] Registered new tserver with Master: 6098e0b8769d4a91aec244bcab43ed81 (127.24.216.130:36965)
I20260929 02:13:36.671201 3287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:46693
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:13:36.705168 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:42755
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:13:37.029368 3615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:13:37.029631 3615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:13:37.029783 3615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:13:37.036960 3615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:13:37.037057 3615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:13:37.037256 3615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:13:37.046725 3615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:42755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:13:37.047857 3615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:13:37.049198 3615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:13:37.062081 3620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:37.062100 3621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:13:37.062132 3623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:13:37.062984 3615 server_base.cc:1034] running on GCE node
I20260929 02:13:37.063524 3615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:13:37.064389 3615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:13:37.065626 3615 hybrid_clock.cc:648] HybridClock initialized: now 1790648017065587 us; error 51 us; skew 500 ppm
I20260929 02:13:37.065950 3615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:13:37 dist-test-slave-9k0h krb5kdc[3242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648017, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:13:37.069123 3615 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:13:37.070588 3615 webserver.cc:460] Webserver started at http://127.24.216.131:39515/ using document root <none> and password file <none>
I20260929 02:13:37.071264 3615 fs_manager.cc:359] Metadata directory not provided
I20260929 02:13:37.071409 3615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:13:37.071707 3615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:13:37.074486 3615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "645f97bf7ecd4a54b67f4f56b2e21179"
format_stamp: "Formatted at 2026-09-29 02:13:37 on dist-test-slave-9k0h"
server_key: "af8e38bdb6a1a9c347a2202a7ce78b74"
server_key_iv: "41f1bec3878efce187b0e140937f28b2"
server_key_version: "encryptionkey@0"
I20260929 02:13:37.075227 3615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "645f97bf7ecd4a54b67f4f56b2e21179"
format_stamp: "Formatted at 2026-09-29 02:13:37 on dist-test-slave-9k0h"
server_key: "af8e38bdb6a1a9c347a2202a7ce78b74"
server_key_iv: "41f1bec3878efce187b0e140937f28b2"
server_key_version: "encryptionkey@0"
I20260929 02:13:37.080139 3615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:13:37.083336 3630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:37.084511 3615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 02:13:37.084684 3615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "645f97bf7ecd4a54b67f4f56b2e21179"
format_stamp: "Formatted at 2026-09-29 02:13:37 on dist-test-slave-9k0h"
server_key: "af8e38bdb6a1a9c347a2202a7ce78b74"
server_key_iv: "41f1bec3878efce187b0e140937f28b2"
server_key_version: "encryptionkey@0"
I20260929 02:13:37.084870 3615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:13:37.117708 3615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:13:37.121933 3615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:13:37.122251 3615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:13:37.123167 3615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:13:37.124449 3615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:13:37.124552 3615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:37.124704 3615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:13:37.124778 3615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:13:37.157506 3615 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:36415
I20260929 02:13:37.157544 3742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:36415 every 8 connection(s)
I20260929 02:13:37.158850 3615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:13:37.167284 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 3615
I20260929 02:13:37.167475 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:13:37.167887 25442 external_mini_cluster.cc:1428] Setting key 85a412979c8b83e96d880a0056cda15e
Sep 29 02:13:37 dist-test-slave-9k0h krb5kdc[3242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648017, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:13:37.178187 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.161198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:56475 (local address 127.24.216.190:42755)
0929 02:13:37.161375 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:37.161381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:37.162527 (+ 1146us) server_negotiation.cc:407] Connection header received
0929 02:13:37.164202 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:37.164207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:37.164267 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:37.164355 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:37.165707 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.166548 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.168314 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.168901 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.169000 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.172894 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:37.172900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:37.172905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:37.172944 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:37.175335 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:37.176029 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:37.176035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:37.176039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:37.176169 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:37.176563 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:37.176570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:37.176573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:37.176811 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:37.176973 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:37.177613 (+ 640us) server_negotiation.cc:299] Negotiation successful
0929 02:13:37.177810 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:13:37.179091 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.161687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:42755 (local address 127.24.216.131:56475)
0929 02:13:37.162381 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:37.162418 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:37.163948 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:37.164555 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:37.164562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:37.165037 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:13:37.165569 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:37.165585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.166680 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.166684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:37.168173 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:37.168185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.169142 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.169150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:37.169433 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.170204 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:13:37.170238 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:13:37.172710 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:13:37.175488 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:37.175496 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:37.175511 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:37.175907 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:37.176273 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:13:37.176278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:13:37.176282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:13:37.176459 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:13:37.176957 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:13:37.176965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:13:37.177281 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 02:13:37.177536 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 02:13:37.178037 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":267,"threads_started":1}
I20260929 02:13:37.180357 3743 heartbeater.cc:347] Connected to a master server at 127.24.216.190:42755
I20260929 02:13:37.180609 3743 heartbeater.cc:464] Registering TS with master...
I20260929 02:13:37.181253 3743 heartbeater.cc:511] Master 127.24.216.190:42755 requested a full tablet report, sending...
I20260929 02:13:37.182785 3287 ts_manager.cc:194] Registered new tserver with Master: 645f97bf7ecd4a54b67f4f56b2e21179 (127.24.216.131:36415)
I20260929 02:13:37.183455 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:13:37.183619 3287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:56475
I20260929 02:13:37.185588 3476 heartbeater.cc:503] Master 127.24.216.190:42755 was elected leader, sending a full tablet report...
I20260929 02:13:37.199898 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.188290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.24.216.190:42755)
0929 02:13:37.188445 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:37.188451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:37.188900 (+ 449us) server_negotiation.cc:407] Connection header received
0929 02:13:37.189063 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:37.189068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:37.189134 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:37.189229 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:37.190647 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.191340 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.192664 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.193082 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.193223 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.195468 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:37.195473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:37.195476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:37.195502 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:37.197762 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:37.198415 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:37.198419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:37.198421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:37.198530 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:37.198823 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:37.198826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:37.198828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:37.199094 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:37.199260 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:37.199539 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 02:13:37.199679 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:13:37.203490 3287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:13:37.209898 3287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:13:37.237293 3755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.222611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:36965 (local address 127.0.0.1:58540)
0929 02:13:37.224365 (+ 1754us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:37.224421 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:37.224632 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:37.226812 (+ 2180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:37.226822 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:37.226875 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:13:37.227420 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:37.227440 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.229793 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.229801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:37.232980 (+ 3179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:37.233000 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.236022 (+ 3022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.236029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:37.236311 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.236384 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 02:13:37.236510 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 02:13:37.236765 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1491,"thread_start_us":1765,"threads_started":1}
I20260929 02:13:37.238948 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.224732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55164 (local address 127.24.216.131:36415)
0929 02:13:37.225464 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:37.225471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:37.227382 (+ 1911us) server_negotiation.cc:407] Connection header received
0929 02:13:37.227638 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:37.227650 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:37.227871 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:37.228055 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:13:37.229249 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.231411 (+ 2162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.234928 (+ 3517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.236952 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.237156 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.237399 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:37.238225 (+ 826us) server_negotiation.cc:299] Negotiation successful
0929 02:13:37.238470 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":425,"threads_started":1}
I20260929 02:13:37.239542 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.222773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.24.216.130:36965)
0929 02:13:37.226268 (+ 3495us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:37.226275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:37.226299 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:13:37.226372 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:37.226378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:37.226587 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:37.226748 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:13:37.227621 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.229605 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.233736 (+ 4131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.235746 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.238615 (+ 2869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.238837 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:37.238931 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 02:13:37.239164 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":972,"thread_start_us":834,"threads_started":1}
I20260929 02:13:37.240048 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.224147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:36415 (local address 127.0.0.1:55164)
0929 02:13:37.227265 (+ 3118us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:37.227279 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:37.227433 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:37.228433 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:37.228441 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:37.228474 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:13:37.228998 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:37.229012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.231687 (+ 2675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.231692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:37.234708 (+ 3016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:37.234728 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.237660 (+ 2932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.237669 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:37.237942 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.237994 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 02:13:37.239546 (+ 1552us) client_negotiation.cc:240] Negotiation successful
0929 02:13:37.239748 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3010,"thread_start_us":2888,"threads_started":1}
I20260929 02:13:37.243676 3545 tablet_service.cc:1467] Processing CreateTablet for tablet 1a089600209946348c5efa53b2efa3c2 (DEFAULT_TABLE table=test-table [id=904bb52e9bd3400c981e5bd6f7cd7a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:37.244629 3678 tablet_service.cc:1467] Processing CreateTablet for tablet 1a089600209946348c5efa53b2efa3c2 (DEFAULT_TABLE table=test-table [id=904bb52e9bd3400c981e5bd6f7cd7a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:37.245158 3545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a089600209946348c5efa53b2efa3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:37.245951 3678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a089600209946348c5efa53b2efa3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:37.246066 3755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.227621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:42749 (local address 127.0.0.1:50738)
0929 02:13:37.238249 (+ 10628us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:37.238272 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:37.238387 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:37.240629 (+ 2242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:37.240632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:37.240652 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:13:37.240823 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:37.240833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.242638 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.242644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:37.243763 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:37.243773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.245356 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.245361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:37.245537 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.245566 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:13:37.245647 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 02:13:37.245785 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10466,"thread_start_us":13208,"threads_started":1}
I20260929 02:13:37.246658 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.227734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.24.216.129:42749)
0929 02:13:37.240054 (+ 12320us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:37.240063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:37.240090 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:13:37.240177 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:37.240184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:37.240443 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:37.240636 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:13:37.241001 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.242495 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.243917 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.244920 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.245106 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.245269 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:37.246305 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0929 02:13:37.246437 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12151,"thread_start_us":11995,"threads_started":1}
I20260929 02:13:37.249217 3411 tablet_service.cc:1467] Processing CreateTablet for tablet 1a089600209946348c5efa53b2efa3c2 (DEFAULT_TABLE table=test-table [id=904bb52e9bd3400c981e5bd6f7cd7a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:13:37.250553 3411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a089600209946348c5efa53b2efa3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:13:37.261631 3763 tablet_bootstrap.cc:492] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01: Bootstrap starting.
I20260929 02:13:37.262554 3761 tablet_bootstrap.cc:492] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81: Bootstrap starting.
I20260929 02:13:37.264920 3763 tablet_bootstrap.cc:654] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:37.265250 3761 tablet_bootstrap.cc:654] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:37.265962 3763 log.cc:824] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01: Log is configured to *not* fsync() on all Append() calls
I20260929 02:13:37.266270 3761 log.cc:824] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81: Log is configured to *not* fsync() on all Append() calls
I20260929 02:13:37.267176 3762 tablet_bootstrap.cc:492] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179: Bootstrap starting.
I20260929 02:13:37.268962 3763 tablet_bootstrap.cc:492] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01: No bootstrap required, opened a new log
I20260929 02:13:37.269224 3763 ts_tablet_manager.cc:1397] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 02:13:37.270087 3762 tablet_bootstrap.cc:654] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179: Neither blocks nor log segments found. Creating new log.
I20260929 02:13:37.271256 3762 log.cc:824] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179: Log is configured to *not* fsync() on all Append() calls
I20260929 02:13:37.272500 3761 tablet_bootstrap.cc:492] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81: No bootstrap required, opened a new log
I20260929 02:13:37.272758 3761 ts_tablet_manager.cc:1397] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20260929 02:13:37.273639 3762 tablet_bootstrap.cc:492] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179: No bootstrap required, opened a new log
I20260929 02:13:37.273874 3762 ts_tablet_manager.cc:1397] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 02:13:37.279368 3763 raft_consensus.cc:348] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6098e0b8769d4a91aec244bcab43ed81" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36965 } } peers { permanent_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36415 } } peers { permanent_uuid: "59b5a7a3f19f4416ab5d0dec36b58a01" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42749 } }
I20260929 02:13:37.279834 3763 raft_consensus.cc:374] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:37.279974 3763 raft_consensus.cc:729] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b5a7a3f19f4416ab5d0dec36b58a01, State: Initialized, Role: FOLLOWER
I20260929 02:13:37.280683 3763 consensus_queue.cc:260] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6098e0b8769d4a91aec244bcab43ed81" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36965 } } peers { permanent_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36415 } } peers { permanent_uuid: "59b5a7a3f19f4416ab5d0dec36b58a01" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42749 } }
I20260929 02:13:37.282491 3763 ts_tablet_manager.cc:1428] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260929 02:13:37.281366 3762 raft_consensus.cc:348] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6098e0b8769d4a91aec244bcab43ed81" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36965 } } peers { permanent_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36415 } } peers { permanent_uuid: "59b5a7a3f19f4416ab5d0dec36b58a01" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42749 } }
I20260929 02:13:37.282841 3762 raft_consensus.cc:374] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:37.282966 3762 raft_consensus.cc:729] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645f97bf7ecd4a54b67f4f56b2e21179, State: Initialized, Role: FOLLOWER
I20260929 02:13:37.283020 3761 raft_consensus.cc:348] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6098e0b8769d4a91aec244bcab43ed81" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36965 } } peers { permanent_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36415 } } peers { permanent_uuid: "59b5a7a3f19f4416ab5d0dec36b58a01" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42749 } }
I20260929 02:13:37.283510 3761 raft_consensus.cc:374] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:13:37.283623 3761 raft_consensus.cc:729] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6098e0b8769d4a91aec244bcab43ed81, State: Initialized, Role: FOLLOWER
I20260929 02:13:37.283540 3762 consensus_queue.cc:260] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6098e0b8769d4a91aec244bcab43ed81" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36965 } } peers { permanent_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36415 } } peers { permanent_uuid: "59b5a7a3f19f4416ab5d0dec36b58a01" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42749 } }
I20260929 02:13:37.284227 3761 consensus_queue.cc:260] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6098e0b8769d4a91aec244bcab43ed81" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36965 } } peers { permanent_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36415 } } peers { permanent_uuid: "59b5a7a3f19f4416ab5d0dec36b58a01" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42749 } }
I20260929 02:13:37.284936 3743 heartbeater.cc:503] Master 127.24.216.190:42755 was elected leader, sending a full tablet report...
I20260929 02:13:37.285516 3762 ts_tablet_manager.cc:1428] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260929 02:13:37.286868 3610 heartbeater.cc:503] Master 127.24.216.190:42755 was elected leader, sending a full tablet report...
I20260929 02:13:37.287292 3761 ts_tablet_manager.cc:1428] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260929 02:13:37.294274 3768 raft_consensus.cc:482] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:13:37.294477 3768 raft_consensus.cc:504] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6098e0b8769d4a91aec244bcab43ed81" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36965 } } peers { permanent_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36415 } } peers { permanent_uuid: "59b5a7a3f19f4416ab5d0dec36b58a01" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42749 } }
I20260929 02:13:37.295912 3768 leader_election.cc:288] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6098e0b8769d4a91aec244bcab43ed81 (127.24.216.130:36965), 59b5a7a3f19f4416ab5d0dec36b58a01 (127.24.216.129:42749)
I20260929 02:13:37.302227 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.296671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:35895 (local address 127.24.216.129:42749)
0929 02:13:37.296860 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:37.296866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:37.296884 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:13:37.296943 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:37.296949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:37.297027 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:37.297111 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:13:37.297481 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.298369 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.300224 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.301280 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.301429 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.301495 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:37.301844 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 02:13:37.301973 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:13:37.302235 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.296228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:42749 (local address 127.24.216.131:35895)
0929 02:13:37.296461 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:37.296484 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:37.296613 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:37.297143 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:37.297147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:37.297175 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:13:37.297353 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:37.297363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.298507 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.298513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:37.300070 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:37.300081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.301414 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.301420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:37.301639 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.301705 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 02:13:37.301801 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 02:13:37.301936 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":73}
I20260929 02:13:37.302400 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.296237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:54845 (local address 127.24.216.130:36965)
0929 02:13:37.296435 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:37.296441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:37.297089 (+ 648us) server_negotiation.cc:407] Connection header received
0929 02:13:37.297189 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:37.297194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:37.297258 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:37.297351 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:13:37.297913 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.298731 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.300472 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.301412 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.301548 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.301639 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:37.302012 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 02:13:37.302133 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:13:37.303057 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.296304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:36965 (local address 127.24.216.131:54845)
0929 02:13:37.296964 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 02:13:37.296979 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:13:37.297098 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:13:37.297375 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:13:37.297378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:13:37.297395 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:13:37.297757 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:37.297768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.298866 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.298869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:37.300289 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:13:37.300301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.301647 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.301652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:13:37.301842 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.301879 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:13:37.302620 (+ 741us) client_negotiation.cc:240] Negotiation successful
0929 02:13:37.302749 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":3,"thread_start_us":381,"threads_started":1}
I20260929 02:13:37.303216 3431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a089600209946348c5efa53b2efa3c2" candidate_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b5a7a3f19f4416ab5d0dec36b58a01" is_pre_election: true
I20260929 02:13:37.303761 3431 raft_consensus.cc:2454] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 645f97bf7ecd4a54b67f4f56b2e21179 in term 0.
I20260929 02:13:37.303870 3565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a089600209946348c5efa53b2efa3c2" candidate_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6098e0b8769d4a91aec244bcab43ed81" is_pre_election: true
I20260929 02:13:37.304334 3565 raft_consensus.cc:2454] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 645f97bf7ecd4a54b67f4f56b2e21179 in term 0.
I20260929 02:13:37.304360 3633 leader_election.cc:302] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59b5a7a3f19f4416ab5d0dec36b58a01, 645f97bf7ecd4a54b67f4f56b2e21179; no voters:
I20260929 02:13:37.304733 3768 raft_consensus.cc:2781] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:13:37.304898 3768 raft_consensus.cc:482] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:13:37.305025 3768 raft_consensus.cc:3037] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:37.307266 3768 raft_consensus.cc:504] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6098e0b8769d4a91aec244bcab43ed81" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36965 } } peers { permanent_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36415 } } peers { permanent_uuid: "59b5a7a3f19f4416ab5d0dec36b58a01" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42749 } }
I20260929 02:13:37.307855 3768 leader_election.cc:288] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [CANDIDATE]: Term 1 election: Requested vote from peers 6098e0b8769d4a91aec244bcab43ed81 (127.24.216.130:36965), 59b5a7a3f19f4416ab5d0dec36b58a01 (127.24.216.129:42749)
I20260929 02:13:37.308194 3565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a089600209946348c5efa53b2efa3c2" candidate_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6098e0b8769d4a91aec244bcab43ed81"
I20260929 02:13:37.308300 3431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a089600209946348c5efa53b2efa3c2" candidate_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b5a7a3f19f4416ab5d0dec36b58a01"
I20260929 02:13:37.308378 3565 raft_consensus.cc:3037] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:37.308521 3431 raft_consensus.cc:3037] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:13:37.310654 3431 raft_consensus.cc:2454] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 645f97bf7ecd4a54b67f4f56b2e21179 in term 1.
I20260929 02:13:37.310665 3565 raft_consensus.cc:2454] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 645f97bf7ecd4a54b67f4f56b2e21179 in term 1.
I20260929 02:13:37.310990 3633 leader_election.cc:302] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59b5a7a3f19f4416ab5d0dec36b58a01, 645f97bf7ecd4a54b67f4f56b2e21179; no voters:
I20260929 02:13:37.311234 3768 raft_consensus.cc:2781] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:13:37.311650 3768 raft_consensus.cc:686] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [term 1 LEADER]: Becoming Leader. State: Replica: 645f97bf7ecd4a54b67f4f56b2e21179, State: Running, Role: LEADER
I20260929 02:13:37.312089 3768 consensus_queue.cc:237] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6098e0b8769d4a91aec244bcab43ed81" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36965 } } peers { permanent_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36415 } } peers { permanent_uuid: "59b5a7a3f19f4416ab5d0dec36b58a01" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42749 } }
I20260929 02:13:37.316673 3286 catalog_manager.cc:5495] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 reported cstate change: term changed from 0 to 1, leader changed from <none> to 645f97bf7ecd4a54b67f4f56b2e21179 (127.24.216.131). New cstate: current_term: 1 leader_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6098e0b8769d4a91aec244bcab43ed81" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "645f97bf7ecd4a54b67f4f56b2e21179" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59b5a7a3f19f4416ab5d0dec36b58a01" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42749 } health_report { overall_health: UNKNOWN } } }
I20260929 02:13:37.348951 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.344197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.24.216.131:36415)
0929 02:13:37.344373 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:37.344378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:37.344469 (+ 91us) server_negotiation.cc:407] Connection header received
0929 02:13:37.344633 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:37.344637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:37.344707 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:37.344832 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:13:37.345146 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.345842 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.347135 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.347517 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.347606 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.347991 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:13:37.348504 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:13:37.348610 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:37.348679 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 02:13:37.348737 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:13:37.357442 3565 raft_consensus.cc:1261] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81 [term 1 FOLLOWER]: Refusing update from remote peer 645f97bf7ecd4a54b67f4f56b2e21179: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:13:37.357453 3431 raft_consensus.cc:1261] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01 [term 1 FOLLOWER]: Refusing update from remote peer 645f97bf7ecd4a54b67f4f56b2e21179: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:13:37.358606 3768 consensus_queue.cc:1035] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6098e0b8769d4a91aec244bcab43ed81" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 36965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:13:37.358969 3771 consensus_queue.cc:1035] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b5a7a3f19f4416ab5d0dec36b58a01" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:13:37.374365 3775 mvcc.cc:205] Tried to move back new op lower bound from 7334494279084642304 to 7334494278914228224. Current Snapshot: MvccSnapshot[applied={T|T < 7334494279084642304}]
I20260929 02:13:37.385443 3287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:13:37.385715 3287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:13:37.389117 3287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 184cd401e7cb42a0a6210e8867ef4b3d: Sending DeleteTablet for 3 replicas of tablet 1a089600209946348c5efa53b2efa3c2
I20260929 02:13:37.390197 3545 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a089600209946348c5efa53b2efa3c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:37 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:58540
I20260929 02:13:37.390300 3678 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a089600209946348c5efa53b2efa3c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:37 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:55164
I20260929 02:13:37.390669 3411 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a089600209946348c5efa53b2efa3c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:13:37 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:50738
I20260929 02:13:37.392956 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 3348
I20260929 02:13:37.403497 3784 tablet_replica.cc:330] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81: stopping tablet replica
I20260929 02:13:37.404047 3784 raft_consensus.cc:2229] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:13:37.404629 3784 raft_consensus.cc:2258] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:37.406813 3784 ts_tablet_manager.cc:1905] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:37.413687 3784 ts_tablet_manager.cc:1918] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:13:37.413903 3784 log.cc:1196] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/1a089600209946348c5efa53b2efa3c2
I20260929 02:13:37.414512 3784 ts_tablet_manager.cc:1939] T 1a089600209946348c5efa53b2efa3c2 P 6098e0b8769d4a91aec244bcab43ed81: Deleting consensus metadata
I20260929 02:13:37.416395 3272 catalog_manager.cc:4841] TS 6098e0b8769d4a91aec244bcab43ed81 (127.24.216.130:36965): tablet 1a089600209946348c5efa53b2efa3c2 (table test-table [id=904bb52e9bd3400c981e5bd6f7cd7a19]) successfully deleted
I20260929 02:13:37.418430 3786 tablet_replica.cc:330] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01: stopping tablet replica
I20260929 02:13:37.418638 3785 tablet_replica.cc:330] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179: stopping tablet replica
I20260929 02:13:37.418862 3786 raft_consensus.cc:2229] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:13:37.419039 3785 raft_consensus.cc:2229] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:13:37.419305 3786 raft_consensus.cc:2258] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:37.419759 3785 raft_consensus.cc:2258] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:13:37.421542 3785 ts_tablet_manager.cc:1905] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:37.422914 3786 ts_tablet_manager.cc:1905] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:13:37.427983 3786 ts_tablet_manager.cc:1918] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:13:37.428244 3786 log.cc:1196] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/1a089600209946348c5efa53b2efa3c2
I20260929 02:13:37.428927 3786 ts_tablet_manager.cc:1939] T 1a089600209946348c5efa53b2efa3c2 P 59b5a7a3f19f4416ab5d0dec36b58a01: Deleting consensus metadata
I20260929 02:13:37.430768 3273 catalog_manager.cc:4841] TS 59b5a7a3f19f4416ab5d0dec36b58a01 (127.24.216.129:42749): tablet 1a089600209946348c5efa53b2efa3c2 (table test-table [id=904bb52e9bd3400c981e5bd6f7cd7a19]) successfully deleted
Sep 29 02:13:37 dist-test-slave-9k0h krb5kdc[3242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:13:37.431826 3785 ts_tablet_manager.cc:1918] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:13:37.432077 3785 log.cc:1196] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/1a089600209946348c5efa53b2efa3c2
I20260929 02:13:37.432650 3785 ts_tablet_manager.cc:1939] T 1a089600209946348c5efa53b2efa3c2 P 645f97bf7ecd4a54b67f4f56b2e21179: Deleting consensus metadata
I20260929 02:13:37.434295 3273 catalog_manager.cc:4841] TS 645f97bf7ecd4a54b67f4f56b2e21179 (127.24.216.131:36415): tablet 1a089600209946348c5efa53b2efa3c2 (table test-table [id=904bb52e9bd3400c981e5bd6f7cd7a19]) successfully deleted
I20260929 02:13:37.437670 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.394389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50746 (local address 127.24.216.129:42749)
0929 02:13:37.394623 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:37.394632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:37.417476 (+ 22844us) server_negotiation.cc:407] Connection header received
0929 02:13:37.417726 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:37.417732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:37.417836 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:37.417961 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:37.421713 (+ 3752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.422937 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.426299 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.426985 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.427075 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.432029 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:37.432045 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:37.432057 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:37.432087 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:37.435069 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:37.435693 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:37.435699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:37.435701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:37.435832 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:37.436221 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:37.436230 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:37.436232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:37.436562 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:37.436858 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:37.437125 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 02:13:37.437331 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:13:37.438321 3471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:37.528805 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 3348
I20260929 02:13:37.546495 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 3482
Sep 29 02:13:37 dist-test-slave-9k0h krb5kdc[3242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:13:37.565126 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.547296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.24.216.130:36965)
0929 02:13:37.547538 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:37.547544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:37.547592 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:13:37.547785 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:37.547791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:37.547891 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:37.548000 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:37.549457 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.550731 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.552480 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.553118 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.553240 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.558389 (+ 5149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:37.558407 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:37.558427 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:37.558468 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:37.562177 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:37.563005 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:37.563014 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:37.563018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:37.563156 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:37.563542 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:37.563547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:37.563550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:37.563993 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:37.564269 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:37.564569 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 02:13:37.564773 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 02:13:37.565770 3605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:37.656520 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 3482
I20260929 02:13:37.680490 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 3615
Sep 29 02:13:37 dist-test-slave-9k0h krb5kdc[3242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:13:37.696810 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:13:37.681362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.24.216.131:36415)
0929 02:13:37.681543 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:13:37.681552 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:13:37.681576 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:13:37.681644 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:13:37.681649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:13:37.681751 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:13:37.681872 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:13:37.683160 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.683990 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.685195 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:13:37.685707 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:13:37.685860 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:13:37.690823 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:13:37.690834 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:13:37.690849 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:13:37.690884 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:13:37.693934 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:37.694717 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:37.694737 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:37.694740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:37.694900 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:13:37.695280 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:13:37.695284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:13:37.695286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:13:37.695619 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:13:37.695969 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0929 02:13:37.696213 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 02:13:37.696422 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:13:37.697484 3738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:37.819283 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 3615
I20260929 02:13:37.839174 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 3255
I20260929 02:13:37.840291 3317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:13:37.947145 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 3255
2026-09-29T02:13:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4132 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:13:38 dist-test-slave-9k0h krb5kdc[3794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:13:38 dist-test-slave-9k0h krb5kdc[3794](info): set up 2 sockets
Sep 29 02:13:38 dist-test-slave-9k0h krb5kdc[3794](info): commencing operation
krb5kdc: starting...
W20260929 02:13:39.063082 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:13:39 dist-test-slave-9k0h krb5kdc[3794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:13:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:13:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.216.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.216.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/rangeradmin_127.24.216.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/rangeradmin_127.24.216.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.216.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.216.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/rangerlookup_127.24.216.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/rangerlookup_127.24.216.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.216.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.216.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/HTTP_127.24.216.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/HTTP_127.24.216.148@KRBTEST.COM.keytab.
I20260929 02:13:39.390340 25442 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGNPMkl/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/postgres -l logfile start
2026-09-29 02:13:41.760 UTC [3824] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 02:13:41.760 UTC [3824] LOG: listening on IPv4 address "127.24.216.148", port 33831
2026-09-29 02:13:41.764 UTC [3824] LOG: listening on Unix socket "/tmp/.s.PGSQL.33831"
2026-09-29 02:13:41.784 UTC [3827] LOG: database system was shut down at 2026-09-29 02:13:40 UTC
2026-09-29 02:13:41.788 UTC [3824] LOG: database system is ready to accept connections
I20260929 02:13:42.802838 25442 mini_postgres.cc:96] Postgres bound to 33831
2026-09-29 02:13:42.822 UTC [3835] FATAL: database "slave" does not exist
127.24.216.148:33831 - accepting connections
I20260929 02:13:42.822783 25442 mini_ranger.cc:161] Starting Ranger...
I20260929 02:13:42.848975 25442 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 02:13:43.295069 25442 mini_ranger.cc:87] Created ranger Postgres database
I20260929 02:13:43.295179 25442 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-admin
2026-09-29 02:13:43,572 [I] DB FLAVOR :POSTGRES
2026-09-29 02:13:43,572 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:13:43,572 [I] Checking connection..
2026-09-29 02:13:43,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 02:13:43,827 [I] Checking connection passed.
2026-09-29 02:13:43,827 [I] --------- Verifying version history table ---------
2026-09-29 02:13:43,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:13:44,084 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:13:44,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:13:44,340 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:13:44,340 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 02:13:44,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 02:13:44.592 UTC [3899] WARNING: there is no transaction in progress
2026-09-29 02:13:44,598 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 02:13:44,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:13:44,829 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 02:13:44,829 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:13:44,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 02:13:45,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 02:13:45,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9k0h.c.gcp-upstream.internal','N') ;"
2026-09-29 02:13:45,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:13:45,808 [I] Table x_portal_user does not exist in database ranger
2026-09-29 02:13:45,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:13:46,062 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 02:13:46,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:13:46,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:13:46,559 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 02:13:46,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 02:13:47.480 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.494 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.509 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.524 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.540 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.557 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.609 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.616 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.630 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.639 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.648 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.661 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.670 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.676 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.682 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:47.688 UTC [4007] WARNING: there is no transaction in progress
2026-09-29 02:13:48,273 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 02:13:48,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 02:13:48,517 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 02:13:48,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:13:48,758 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 02:13:48,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:13:49,011 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 02:13:49,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:13:49,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:13:49,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9k0h.c.gcp-upstream.internal';"
2026-09-29 02:13:49,748 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 02:13:49,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 02:13:50,001 [I] DB_PATCHES have already been applied
I20260929 02:13:50.007647 25442 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGNPMkl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 02:13:50.007771 25442 mini_ranger.cc:193] Using host: 127.24.216.148
I20260929 02:13:50.024479 25442 mini_ranger.cc:227] Ranger admin URL: http://127.24.216.148:47285
Sep 29, 2026 2:13:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:13:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:13:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:13:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:13:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:13:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:13:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 02:13:51.116518 25442 mini_ranger.cc:160] Time spent starting Ranger: real 8.294s user 0.001s sys 0.044s
Sep 29, 2026 2:14:10 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:14:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 2:14:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:14:10 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:14:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:14:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:14:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:14:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:14:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:14:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:14:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:14:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:14:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:14:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:14:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:14:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:14:10 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:14:11 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 02:14:11.822309 25442 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.216.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.216.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/rangerkms_127.24.216.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/rangerkms_127.24.216.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.216.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.216.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.216.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/HTTP_127.24.216.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.216.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/HTTP_127.24.216.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 02:14:11 dist-test-slave-9k0h krb5kdc[3794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648051, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 02:14:11.988622 25442 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 02:14:12.008075 25442 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 02:14:12.793946 25442 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 02:14:12.794127 25442 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-kms
I20260929 02:14:12.794332 25442 mini_ranger_kms.cc:227] Using postgres at 127.24.216.148:33831
2026-09-29 02:14:13,240 [I] DB FLAVOR :POSTGRES
2026-09-29 02:14:13,240 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:14:13,240 [I] Checking connection
2026-09-29 02:14:13,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 02:14:13,460 [I] connection success
2026-09-29 02:14:13,460 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 02:14:13,460 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 02:14:13,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 02:14:13,698 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 02:14:13,699 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:14:13,699 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 02:14:13,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:33831/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 02:14:13,977 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 02:14:14.403177 25442 mini_ranger_kms.cc:310] Created kms service
I20260929 02:14:14.572625 25442 mini_ranger_kms.cc:326] Created kudu user
I20260929 02:14:14.617156 25442 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 02:14:14 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648051, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
I20260929 02:14:14.997002 25442 mini_ranger_kms.cc:384] Added ranger policy
I20260929 02:14:14.997339 25442 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 02:14:14.997398 25442 mini_ranger_kms.cc:240] Using host: 127.24.216.148
I20260929 02:14:15.019075 25442 mini_ranger_kms.cc:276] Ranger KMS PID: 4187
I20260929 02:14:15.019236 25442 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.216.148:50603
Sep 29, 2026 2:14:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:14:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:14:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:14:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:14:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:14:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 02:14:17 dist-test-slave-9k0h krb5kdc[3794](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790648057, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.216.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:14:17.261623 25442 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.273s user 0.126s sys 0.341s
I20260929 02:14:17.261904 25442 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 02:14:17.262137 25442 mini_ranger_kms.cc:401] 127.24.216.148:50603/kms/v1/keys
Sep 29 02:14:17 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648057, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.216.148@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
Sep 29 02:14:18 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648057, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.216.148@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
Sep 29 02:14:20 dist-test-slave-9k0h krb5kdc[3794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:14:20.306351 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:45139
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:45139
--ranger_config_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.216.148:50603/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:14:20.680012 4238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:14:20.680270 4238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:14:20.680352 4238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:14:20.687513 4238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:14:20.687616 4238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:14:20.687668 4238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:14:20.687722 4238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:14:20.687772 4238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:14:20.698594 4238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.216.148:50603/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:45139
--ranger_config_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:45139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:14:20.699800 4238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:14:20.701032 4238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:14:20.713581 4246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:14:20.713657 4243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:14:20.713631 4244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:14:20.714895 4238 server_base.cc:1034] running on GCE node
I20260929 02:14:20.715590 4238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:14:20.716943 4238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:14:20.718210 4238 hybrid_clock.cc:648] HybridClock initialized: now 1790648060718126 us; error 54 us; skew 500 ppm
I20260929 02:14:20.718550 4238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:14:20 dist-test-slave-9k0h krb5kdc[3794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648060, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:14:20.730330 4238 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:14:20.731792 4238 webserver.cc:460] Webserver started at http://127.24.216.190:36485/ using document root <none> and password file <none>
I20260929 02:14:20.732533 4238 fs_manager.cc:359] Metadata directory not provided
I20260929 02:14:20.732676 4238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:14:20.732959 4238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:14:20 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648060, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
I20260929 02:14:20.784663 4238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "faefc1e5eb9b421f8de1c444376b2473"
format_stamp: "Formatted at 2026-09-29 02:14:20 on dist-test-slave-9k0h"
server_key: "1c7de4f35b26a3f98ea2f0eb6937f784"
server_key_iv: "9ac0cad6f4ac29043c23dd9b1402c446"
server_key_version: "kuduclusterkey@0"
I20260929 02:14:20.785519 4238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "faefc1e5eb9b421f8de1c444376b2473"
format_stamp: "Formatted at 2026-09-29 02:14:20 on dist-test-slave-9k0h"
server_key: "1c7de4f35b26a3f98ea2f0eb6937f784"
server_key_iv: "9ac0cad6f4ac29043c23dd9b1402c446"
server_key_version: "kuduclusterkey@0"
I20260929 02:14:20.791281 4238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:14:20.809808 4253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:14:20.811393 4238 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:14:20.811579 4238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "faefc1e5eb9b421f8de1c444376b2473"
format_stamp: "Formatted at 2026-09-29 02:14:20 on dist-test-slave-9k0h"
server_key: "1c7de4f35b26a3f98ea2f0eb6937f784"
server_key_iv: "9ac0cad6f4ac29043c23dd9b1402c446"
server_key_version: "kuduclusterkey@0"
I20260929 02:14:20.811781 4238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:14:20.848927 4238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:14:20.853066 4238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:14:20.853384 4238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:14:20.873531 4238 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:45139
I20260929 02:14:20.873572 4304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:45139 every 8 connection(s)
I20260929 02:14:20.874907 4238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:14:20.878652 4305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:14:20.879899 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 4238
I20260929 02:14:20.880112 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:14:20.880734 25442 external_mini_cluster.cc:1428] Setting key 3657ced9710c89d3a488dac1431dddae
I20260929 02:14:20.892688 4305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473: Bootstrap starting.
Sep 29 02:14:20 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:14:20.896263 4305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473: Neither blocks nor log segments found. Creating new log.
I20260929 02:14:20.897397 4305 log.cc:824] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473: Log is configured to *not* fsync() on all Append() calls
I20260929 02:14:20.900239 4305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473: No bootstrap required, opened a new log
I20260929 02:14:20.902920 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:20.882659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54296 (local address 127.24.216.190:45139)
0929 02:14:20.883431 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0929 02:14:20.883442 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:14:20.883484 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:14:20.884961 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:14:20.884983 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:14:20.885368 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:14:20.885683 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:14:20.887215 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:20.888469 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:20.890750 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:20.891405 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:20.891536 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:20.895895 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:14:20.895903 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:14:20.895916 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:14:20.895948 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:14:20.899100 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:20.899773 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:20.899779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:20.899785 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:20.899926 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:20.900284 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:20.900293 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:20.900295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:20.900801 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:14:20.901084 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 02:14:20.901554 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 02:14:20.901911 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":372,"threads_started":1}
I20260929 02:14:20.909436 4305 raft_consensus.cc:348] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faefc1e5eb9b421f8de1c444376b2473" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45139 } }
I20260929 02:14:20.909806 4305 raft_consensus.cc:374] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:14:20.909907 4305 raft_consensus.cc:729] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faefc1e5eb9b421f8de1c444376b2473, State: Initialized, Role: FOLLOWER
I20260929 02:14:20.910557 4305 consensus_queue.cc:260] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faefc1e5eb9b421f8de1c444376b2473" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45139 } }
I20260929 02:14:20.910852 4305 raft_consensus.cc:388] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:14:20.910997 4305 raft_consensus.cc:482] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:14:20.911186 4305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:14:20.913205 4305 raft_consensus.cc:504] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faefc1e5eb9b421f8de1c444376b2473" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45139 } }
I20260929 02:14:20.913672 4305 leader_election.cc:302] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faefc1e5eb9b421f8de1c444376b2473; no voters:
I20260929 02:14:20.914408 4305 leader_election.cc:288] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:14:20.914482 4310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:14:20.915242 4310 raft_consensus.cc:686] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [term 1 LEADER]: Becoming Leader. State: Replica: faefc1e5eb9b421f8de1c444376b2473, State: Running, Role: LEADER
I20260929 02:14:20.915797 4310 consensus_queue.cc:237] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faefc1e5eb9b421f8de1c444376b2473" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45139 } }
I20260929 02:14:20.916431 4305 sys_catalog.cc:564] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:14:20.919958 4312 sys_catalog.cc:455] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faefc1e5eb9b421f8de1c444376b2473. Latest consensus state: current_term: 1 leader_uuid: "faefc1e5eb9b421f8de1c444376b2473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faefc1e5eb9b421f8de1c444376b2473" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45139 } } }
I20260929 02:14:20.920297 4312 sys_catalog.cc:458] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [sys.catalog]: This master's current role is: LEADER
I20260929 02:14:20.920775 4311 sys_catalog.cc:455] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faefc1e5eb9b421f8de1c444376b2473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faefc1e5eb9b421f8de1c444376b2473" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45139 } } }
I20260929 02:14:20.921041 4311 sys_catalog.cc:458] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473 [sys.catalog]: This master's current role is: LEADER
I20260929 02:14:20.922415 4319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:14:20.927446 4319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:14:20.928213 4305 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 02:14:20.946766 4319 catalog_manager.cc:1347] Generated new cluster ID: 42454ba862b14efba69930e9c36d91eb
I20260929 02:14:20.947027 4319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:14:20.970777 4319 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:14:20.972201 4319 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:14:20.987422 4319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473: Generated new TSK 0
I20260929 02:14:20.988672 4319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:14:21.758255 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.216.148:50603/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:45139
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:14:22 dist-test-slave-9k0h krb5kdc[3794](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790648062, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 02:14:22.181517 4350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:14:22.181855 4350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:14:22.182034 4350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:14:22.190589 4350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:14:22.190729 4350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:14:22.190925 4350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:14:22.200836 4350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.216.148:50603/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:45139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:14:22.202365 4350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:14:22.203673 4350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:14:22.222131 4360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:14:22.224229 4361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:14:22.228365 4350 server_base.cc:1034] running on GCE node
W20260929 02:14:22.228964 4363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:14:22.229686 4350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:14:22.230783 4350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:14:22.232055 4350 hybrid_clock.cc:648] HybridClock initialized: now 1790648062231996 us; error 45 us; skew 500 ppm
I20260929 02:14:22.232399 4350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:14:22 dist-test-slave-9k0h krb5kdc[3794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648062, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:14:22.236668 4350 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:14:22.239894 4350 webserver.cc:460] Webserver started at http://127.24.216.129:35273/ using document root <none> and password file <none>
I20260929 02:14:22.240619 4350 fs_manager.cc:359] Metadata directory not provided
I20260929 02:14:22.240769 4350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:14:22.241086 4350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:14:22 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648062, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
I20260929 02:14:22.260362 4350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "de177e9eae86499eaf8fc793a688eed3"
format_stamp: "Formatted at 2026-09-29 02:14:22 on dist-test-slave-9k0h"
server_key: "5b8fb64e22e0a2c9a323e2239ab1a6d6"
server_key_iv: "ab5b49ae1ad81fd39ba2311b6d53880b"
server_key_version: "kuduclusterkey@0"
I20260929 02:14:22.261786 4350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "de177e9eae86499eaf8fc793a688eed3"
format_stamp: "Formatted at 2026-09-29 02:14:22 on dist-test-slave-9k0h"
server_key: "5b8fb64e22e0a2c9a323e2239ab1a6d6"
server_key_iv: "ab5b49ae1ad81fd39ba2311b6d53880b"
server_key_version: "kuduclusterkey@0"
I20260929 02:14:22.280104 4350 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.000s sys 0.015s
I20260929 02:14:22.309798 4370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:14:22.317234 4350 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.008s sys 0.006s
I20260929 02:14:22.317540 4350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "de177e9eae86499eaf8fc793a688eed3"
format_stamp: "Formatted at 2026-09-29 02:14:22 on dist-test-slave-9k0h"
server_key: "5b8fb64e22e0a2c9a323e2239ab1a6d6"
server_key_iv: "ab5b49ae1ad81fd39ba2311b6d53880b"
server_key_version: "kuduclusterkey@0"
I20260929 02:14:22.317832 4350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:14:22.350718 4350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:14:22.354964 4350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:14:22.355273 4350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:14:22.358822 4350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:14:22.360232 4350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:14:22.360337 4350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:14:22.360492 4350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:14:22.360565 4350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:14:22.431030 4350 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:33033
I20260929 02:14:22.431083 4482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:33033 every 8 connection(s)
I20260929 02:14:22.432649 4350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:14:22.435475 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 4350
I20260929 02:14:22.435693 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:14:22.436517 25442 external_mini_cluster.cc:1428] Setting key 71a59c6408ca88e38909c809b09b8cfc
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Sep 29 02:14:22 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648062, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:14:22.512562 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:22.439439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:46651 (local address 127.24.216.190:45139)
0929 02:14:22.461168 (+ 21729us) server_negotiation.cc:206] Beginning negotiation
0929 02:14:22.461176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:14:22.483254 (+ 22078us) server_negotiation.cc:407] Connection header received
0929 02:14:22.486466 (+ 3212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:14:22.486473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:14:22.486656 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:14:22.486796 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:14:22.489620 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:22.490999 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:22.498273 (+ 7274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:22.498915 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:22.499047 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:22.504587 (+ 5540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:14:22.504594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:14:22.504599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:14:22.504638 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:14:22.508527 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:22.509507 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:22.509516 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:22.509522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:22.509672 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:22.510245 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:22.510252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:22.510258 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:22.510665 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:14:22.510806 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 02:14:22.511835 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0929 02:14:22.512100 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21580,"thread_start_us":22741,"threads_started":1}
I20260929 02:14:22.515516 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:22.438614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:45139 (local address 127.24.216.129:46651)
0929 02:14:22.482340 (+ 43726us) negotiation.cc:107] Waiting for socket to connect
0929 02:14:22.482417 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 02:14:22.486095 (+ 3678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:14:22.487182 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:14:22.487195 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:14:22.488094 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:14:22.489407 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:22.489441 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:22.492971 (+ 3530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:22.492978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:22.495531 (+ 2553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:22.495546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:22.499368 (+ 3822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:22.499377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:22.499724 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:22.501290 (+ 1566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:14:22.501320 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:14:22.504335 (+ 3015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:14:22.508698 (+ 4363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:14:22.508710 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:14:22.508750 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:14:22.509359 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:14:22.509821 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:14:22.509828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:14:22.509834 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:14:22.510087 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:14:22.510923 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:14:22.510934 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:14:22.511412 (+ 478us) client_negotiation.cc:769] Sending connection context
0929 02:14:22.513116 (+ 1704us) client_negotiation.cc:240] Negotiation successful
0929 02:14:22.513773 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34214,"thread_start_us":34530,"threads_started":1}
I20260929 02:14:22.517618 4483 heartbeater.cc:347] Connected to a master server at 127.24.216.190:45139
I20260929 02:14:22.517963 4483 heartbeater.cc:464] Registering TS with master...
I20260929 02:14:22.518827 4483 heartbeater.cc:511] Master 127.24.216.190:45139 requested a full tablet report, sending...
I20260929 02:14:22.521736 4270 ts_manager.cc:194] Registered new tserver with Master: de177e9eae86499eaf8fc793a688eed3 (127.24.216.129:33033)
I20260929 02:14:22.523888 4270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:46651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:14:22.533299 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.216.148:50603/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:45139
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:14:22 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648062, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
Sep 29 02:14:22 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648062, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
W20260929 02:14:23.095714 4489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:14:23.096385 4489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:14:23.096818 4489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:14:23.110725 4489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:14:23.110949 4489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:14:23.111306 4489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:14:23.129276 4489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.216.148:50603/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:45139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:14:23.131592 4489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:14:23.133517 4489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:14:23.165385 4497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:14:23.170435 4496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:14:23.178571 4489 server_base.cc:1034] running on GCE node
W20260929 02:14:23.178063 4499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:14:23.179754 4489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:14:23.182441 4489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:14:23.183768 4489 hybrid_clock.cc:648] HybridClock initialized: now 1790648063183646 us; error 106 us; skew 500 ppm
I20260929 02:14:23.184165 4489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:14:23 dist-test-slave-9k0h krb5kdc[3794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648063, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:14:23.189298 4489 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:14:23.191617 4489 webserver.cc:460] Webserver started at http://127.24.216.130:33517/ using document root <none> and password file <none>
I20260929 02:14:23.192539 4489 fs_manager.cc:359] Metadata directory not provided
I20260929 02:14:23.192688 4489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:14:23.193079 4489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:14:23 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648063, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
I20260929 02:14:23.213953 4489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "6477a5027a9145b3bbcf4cab8b90e896"
format_stamp: "Formatted at 2026-09-29 02:14:23 on dist-test-slave-9k0h"
server_key: "8779d19fbb45ee0ae6c3f09b3cf96401"
server_key_iv: "87efb75312b1d627e7a268fb6ece24f9"
server_key_version: "kuduclusterkey@0"
I20260929 02:14:23.217908 4489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "6477a5027a9145b3bbcf4cab8b90e896"
format_stamp: "Formatted at 2026-09-29 02:14:23 on dist-test-slave-9k0h"
server_key: "8779d19fbb45ee0ae6c3f09b3cf96401"
server_key_iv: "87efb75312b1d627e7a268fb6ece24f9"
server_key_version: "kuduclusterkey@0"
I20260929 02:14:23.227180 4489 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260929 02:14:23.248802 4513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:14:23.250427 4489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:14:23.250604 4489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "6477a5027a9145b3bbcf4cab8b90e896"
format_stamp: "Formatted at 2026-09-29 02:14:23 on dist-test-slave-9k0h"
server_key: "8779d19fbb45ee0ae6c3f09b3cf96401"
server_key_iv: "87efb75312b1d627e7a268fb6ece24f9"
server_key_version: "kuduclusterkey@0"
I20260929 02:14:23.250833 4489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:14:23.276861 4489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:14:23.281250 4489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:14:23.281569 4489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:14:23.282673 4489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:14:23.284153 4489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:14:23.284255 4489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:14:23.284394 4489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:14:23.284467 4489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:14:23.342724 4489 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:38063
I20260929 02:14:23.342851 4625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:38063 every 8 connection(s)
I20260929 02:14:23.344081 4489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:14:23.349025 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 4489
I20260929 02:14:23.349193 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:14:23.349701 25442 external_mini_cluster.cc:1428] Setting key ad53fbb5916fc420cce9dab116d34e2b
Sep 29 02:14:23 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648063, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:14:23.370040 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:23.346788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:56535 (local address 127.24.216.190:45139)
0929 02:14:23.347397 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0929 02:14:23.347404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:14:23.348469 (+ 1065us) server_negotiation.cc:407] Connection header received
0929 02:14:23.350873 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:14:23.350878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:14:23.350955 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:14:23.351080 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:14:23.353281 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:23.354394 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:23.357386 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:23.357994 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:23.358122 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:23.362567 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:14:23.362574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:14:23.362579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:14:23.362617 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:14:23.366428 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:23.367345 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:23.367351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:23.367355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:23.367501 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:23.368007 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:23.368012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:23.368015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:23.368374 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:14:23.368512 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 02:14:23.369405 (+ 893us) server_negotiation.cc:299] Negotiation successful
0929 02:14:23.369629 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":332,"threads_started":1}
I20260929 02:14:23.371440 4629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:23.347426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:45139 (local address 127.24.216.130:56535)
0929 02:14:23.348292 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0929 02:14:23.348351 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 02:14:23.350355 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:14:23.351361 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:14:23.351372 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:14:23.352173 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:14:23.353094 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:23.353121 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:23.354601 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:23.354609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:23.357217 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:23.357235 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:23.358254 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:23.358264 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:23.358599 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:23.359416 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:14:23.359454 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:14:23.362280 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:14:23.366573 (+ 4293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:14:23.366584 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:14:23.366606 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:14:23.367212 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:14:23.367626 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:14:23.367634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:14:23.367641 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:14:23.367887 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:14:23.368507 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:14:23.368518 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:14:23.368967 (+ 449us) client_negotiation.cc:769] Sending connection context
0929 02:14:23.369320 (+ 353us) client_negotiation.cc:240] Negotiation successful
0929 02:14:23.369884 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":393,"threads_started":1}
I20260929 02:14:23.373028 4626 heartbeater.cc:347] Connected to a master server at 127.24.216.190:45139
I20260929 02:14:23.373350 4626 heartbeater.cc:464] Registering TS with master...
I20260929 02:14:23.374059 4626 heartbeater.cc:511] Master 127.24.216.190:45139 requested a full tablet report, sending...
I20260929 02:14:23.375481 4270 ts_manager.cc:194] Registered new tserver with Master: 6477a5027a9145b3bbcf4cab8b90e896 (127.24.216.130:38063)
I20260929 02:14:23.376363 4270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:56535
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:14:23.430266 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.216.148:50603/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:45139
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:14:23.527577 4483 heartbeater.cc:503] Master 127.24.216.190:45139 was elected leader, sending a full tablet report...
W20260929 02:14:23.816211 4632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:14:23.816535 4632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:14:23.816708 4632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:14:23.825240 4632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:14:23.825392 4632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:14:23.825587 4632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:14:23.835743 4632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.216.148:50603/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:45139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:14:23.837265 4632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:14:23.838814 4632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:14:23.852434 4637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:14:23.852349 4640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:14:23.852521 4638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:14:23.858549 4632 server_base.cc:1034] running on GCE node
I20260929 02:14:23.859207 4632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:14:23.860234 4632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:14:23.861478 4632 hybrid_clock.cc:648] HybridClock initialized: now 1790648063861410 us; error 50 us; skew 500 ppm
I20260929 02:14:23.861930 4632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:14:23 dist-test-slave-9k0h krb5kdc[3794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648063, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:14:23.865938 4632 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:14:23.867451 4632 webserver.cc:460] Webserver started at http://127.24.216.131:36781/ using document root <none> and password file <none>
I20260929 02:14:23.868103 4632 fs_manager.cc:359] Metadata directory not provided
I20260929 02:14:23.868199 4632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:14:23.868485 4632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:14:23 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648063, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
I20260929 02:14:23.885201 4632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "bbdbef92d57b42cba823bbed27dfd341"
format_stamp: "Formatted at 2026-09-29 02:14:23 on dist-test-slave-9k0h"
server_key: "d1d3da149b4800a050cf66d5b0ffa0c0"
server_key_iv: "e608efde09ae9ccb29ff90e9c99e3bd3"
server_key_version: "kuduclusterkey@0"
I20260929 02:14:23.886286 4632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "bbdbef92d57b42cba823bbed27dfd341"
format_stamp: "Formatted at 2026-09-29 02:14:23 on dist-test-slave-9k0h"
server_key: "d1d3da149b4800a050cf66d5b0ffa0c0"
server_key_iv: "e608efde09ae9ccb29ff90e9c99e3bd3"
server_key_version: "kuduclusterkey@0"
I20260929 02:14:23.892119 4632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:14:23.909092 4647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:14:23.910597 4632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:14:23.910737 4632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "bbdbef92d57b42cba823bbed27dfd341"
format_stamp: "Formatted at 2026-09-29 02:14:23 on dist-test-slave-9k0h"
server_key: "d1d3da149b4800a050cf66d5b0ffa0c0"
server_key_iv: "e608efde09ae9ccb29ff90e9c99e3bd3"
server_key_version: "kuduclusterkey@0"
I20260929 02:14:23.910912 4632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:14:23.932004 4632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:14:23.935781 4632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:14:23.936060 4632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:14:23.936892 4632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:14:23.938104 4632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:14:23.938205 4632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:14:23.938356 4632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:14:23.938401 4632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:14:23.974452 4632 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:38901
I20260929 02:14:23.974591 4759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:38901 every 8 connection(s)
I20260929 02:14:23.975811 4632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:14:23.985600 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 4632
I20260929 02:14:23.985821 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:14:23.986384 25442 external_mini_cluster.cc:1428] Setting key fbf9f03eb1622a8a7ae54cff9ad58aea
Sep 29 02:14:23 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648063, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:14:23.995973 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:23.978649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:35189 (local address 127.24.216.190:45139)
0929 02:14:23.979096 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 02:14:23.979100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:14:23.980076 (+ 976us) server_negotiation.cc:407] Connection header received
0929 02:14:23.981770 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:14:23.981774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:14:23.981860 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:14:23.981956 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:14:23.983372 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:23.984198 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:23.986101 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:23.986656 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:23.986768 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:23.990433 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:14:23.990439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:14:23.990450 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:14:23.990481 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:14:23.993012 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:23.993709 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:23.993716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:23.993720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:23.993844 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:23.994322 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:23.994329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:23.994332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:23.994606 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:14:23.994755 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 02:14:23.995408 (+ 653us) server_negotiation.cc:299] Negotiation successful
0929 02:14:23.995627 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":223,"threads_started":1}
I20260929 02:14:23.996894 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:23.979228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:45139 (local address 127.24.216.131:35189)
0929 02:14:23.979912 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 02:14:23.979954 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:14:23.981537 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:14:23.982235 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:14:23.982245 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:14:23.982644 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:14:23.983198 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:23.983214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:23.984333 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:23.984336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:23.985953 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:23.985969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:23.986901 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:23.986924 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:23.987185 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:23.987855 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:14:23.987875 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:14:23.990230 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:14:23.993158 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:14:23.993166 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:14:23.993182 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:14:23.993591 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:14:23.993953 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:14:23.993957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:14:23.993961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:14:23.994218 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:14:23.994730 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:14:23.994738 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:14:23.995059 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 02:14:23.995317 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 02:14:23.995790 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":279,"threads_started":1}
I20260929 02:14:23.998273 4760 heartbeater.cc:347] Connected to a master server at 127.24.216.190:45139
I20260929 02:14:23.998533 4760 heartbeater.cc:464] Registering TS with master...
I20260929 02:14:23.999138 4760 heartbeater.cc:511] Master 127.24.216.190:45139 requested a full tablet report, sending...
I20260929 02:14:24.000488 4270 ts_manager.cc:194] Registered new tserver with Master: bbdbef92d57b42cba823bbed27dfd341 (127.24.216.131:38901)
I20260929 02:14:24.001441 4270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:35189
I20260929 02:14:24.001854 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:14:24.018342 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.006275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.24.216.190:45139)
0929 02:14:24.006485 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:14:24.006494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:14:24.006868 (+ 374us) server_negotiation.cc:407] Connection header received
0929 02:14:24.006982 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:14:24.006985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:14:24.007059 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:14:24.007136 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:14:24.008488 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.009283 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.010628 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.011026 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.011116 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.013296 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:14:24.013300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:14:24.013303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:14:24.013325 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:14:24.016005 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:24.016673 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:24.016677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:24.016679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:24.016788 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:24.017092 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:24.017096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:24.017097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:24.017384 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:14:24.017633 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 02:14:24.017838 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 02:14:24.017982 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:14:24.022068 4270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:14:24.029173 4270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:14:24.051574 4774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.041573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.24.216.130:38063)
0929 02:14:24.042700 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0929 02:14:24.042706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:14:24.042727 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:14:24.042800 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:14:24.042805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:14:24.043081 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:14:24.043255 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:14:24.044409 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.046351 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.049366 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.050414 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.050578 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.050742 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 02:14:24.051097 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 02:14:24.051258 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":794,"threads_started":1}
I20260929 02:14:24.051647 4773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.041314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:38063 (local address 127.0.0.1:45800)
0929 02:14:24.042269 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0929 02:14:24.042323 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:14:24.042489 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:14:24.043575 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:14:24.043586 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:14:24.043626 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:14:24.044177 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:24.044194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.046538 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.046545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:24.049221 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:24.049231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.050609 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.050615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:24.050842 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.050891 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 02:14:24.051024 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 02:14:24.051216 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":67,"thread_start_us":524,"threads_started":1}
I20260929 02:14:24.052753 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.042794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:38901 (local address 127.0.0.1:38474)
0929 02:14:24.043373 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 02:14:24.043389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:14:24.043492 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:14:24.045687 (+ 2195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:14:24.045694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:14:24.045719 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:14:24.046075 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:24.046089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.047754 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.047762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:24.050276 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:24.050291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.051998 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.052004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:24.052163 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.052217 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 02:14:24.052280 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 02:14:24.052388 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":50,"thread_start_us":316,"threads_started":1}
I20260929 02:14:24.053821 4777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.043455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:33033 (local address 127.0.0.1:35516)
0929 02:14:24.044027 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 02:14:24.044048 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:14:24.044165 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:14:24.046283 (+ 2118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:14:24.046291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:14:24.046316 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:14:24.046748 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:24.046758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.048986 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.048992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:24.051254 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:24.051266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.053066 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.053072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:24.053219 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.053255 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 02:14:24.053337 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 02:14:24.053457 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":3,"thread_start_us":724,"threads_started":1}
I20260929 02:14:24.053848 4778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.044007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.24.216.129:33033)
0929 02:14:24.045754 (+ 1747us) server_negotiation.cc:206] Beginning negotiation
0929 02:14:24.045762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:14:24.045787 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:14:24.045880 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:14:24.045887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:14:24.046081 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:14:24.046934 (+ 853us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:14:24.047263 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.048856 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.051438 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.052883 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.053072 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.053224 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 02:14:24.053381 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 02:14:24.053520 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1596,"thread_start_us":1492,"threads_started":1}
I20260929 02:14:24.054158 4776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.043115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38474 (local address 127.24.216.131:38901)
0929 02:14:24.045049 (+ 1934us) server_negotiation.cc:206] Beginning negotiation
0929 02:14:24.045061 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:14:24.045103 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:14:24.045221 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:14:24.045231 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:14:24.045482 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:14:24.046297 (+ 815us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:14:24.046604 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.047621 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.050444 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.051800 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.053286 (+ 1486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.053435 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 02:14:24.053510 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 02:14:24.053655 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1715,"thread_start_us":1502,"threads_started":1}
I20260929 02:14:24.055724 4561 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1ece05a53e483fb47dffc1468f9c0c (DEFAULT_TABLE table=test-table [id=ad000645ad59469d8a7bec9c39ef6b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:14:24.056995 4561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1ece05a53e483fb47dffc1468f9c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:14:24.057560 4418 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1ece05a53e483fb47dffc1468f9c0c (DEFAULT_TABLE table=test-table [id=ad000645ad59469d8a7bec9c39ef6b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:14:24.057566 4695 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1ece05a53e483fb47dffc1468f9c0c (DEFAULT_TABLE table=test-table [id=ad000645ad59469d8a7bec9c39ef6b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:14:24.058954 4418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1ece05a53e483fb47dffc1468f9c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:14:24.058947 4695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1ece05a53e483fb47dffc1468f9c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:14:24.067764 4779 tablet_bootstrap.cc:492] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896: Bootstrap starting.
I20260929 02:14:24.071163 4779 tablet_bootstrap.cc:654] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896: Neither blocks nor log segments found. Creating new log.
I20260929 02:14:24.072124 4779 log.cc:824] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896: Log is configured to *not* fsync() on all Append() calls
I20260929 02:14:24.073182 4781 tablet_bootstrap.cc:492] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3: Bootstrap starting.
I20260929 02:14:24.074103 4780 tablet_bootstrap.cc:492] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341: Bootstrap starting.
I20260929 02:14:24.078929 4780 tablet_bootstrap.cc:654] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341: Neither blocks nor log segments found. Creating new log.
I20260929 02:14:24.080502 4780 log.cc:824] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341: Log is configured to *not* fsync() on all Append() calls
I20260929 02:14:24.081691 4781 tablet_bootstrap.cc:654] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3: Neither blocks nor log segments found. Creating new log.
I20260929 02:14:24.082640 4779 tablet_bootstrap.cc:492] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896: No bootstrap required, opened a new log
I20260929 02:14:24.082904 4781 log.cc:824] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:14:24.082959 4779 ts_tablet_manager.cc:1397] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.006s
I20260929 02:14:24.083575 4780 tablet_bootstrap.cc:492] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341: No bootstrap required, opened a new log
I20260929 02:14:24.083880 4780 ts_tablet_manager.cc:1397] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 02:14:24.085932 4781 tablet_bootstrap.cc:492] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3: No bootstrap required, opened a new log
I20260929 02:14:24.086210 4781 ts_tablet_manager.cc:1397] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260929 02:14:24.092340 4780 raft_consensus.cc:348] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6477a5027a9145b3bbcf4cab8b90e896" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38063 } } peers { permanent_uuid: "bbdbef92d57b42cba823bbed27dfd341" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38901 } } peers { permanent_uuid: "de177e9eae86499eaf8fc793a688eed3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33033 } }
I20260929 02:14:24.092705 4780 raft_consensus.cc:374] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:14:24.092823 4780 raft_consensus.cc:729] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdbef92d57b42cba823bbed27dfd341, State: Initialized, Role: FOLLOWER
I20260929 02:14:24.093401 4780 consensus_queue.cc:260] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6477a5027a9145b3bbcf4cab8b90e896" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38063 } } peers { permanent_uuid: "bbdbef92d57b42cba823bbed27dfd341" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38901 } } peers { permanent_uuid: "de177e9eae86499eaf8fc793a688eed3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33033 } }
I20260929 02:14:24.094591 4760 heartbeater.cc:503] Master 127.24.216.190:45139 was elected leader, sending a full tablet report...
I20260929 02:14:24.095250 4780 ts_tablet_manager.cc:1428] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 02:14:24.096666 4779 raft_consensus.cc:348] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6477a5027a9145b3bbcf4cab8b90e896" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38063 } } peers { permanent_uuid: "bbdbef92d57b42cba823bbed27dfd341" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38901 } } peers { permanent_uuid: "de177e9eae86499eaf8fc793a688eed3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33033 } }
I20260929 02:14:24.097296 4779 raft_consensus.cc:374] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:14:24.097461 4779 raft_consensus.cc:729] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6477a5027a9145b3bbcf4cab8b90e896, State: Initialized, Role: FOLLOWER
I20260929 02:14:24.097354 4781 raft_consensus.cc:348] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6477a5027a9145b3bbcf4cab8b90e896" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38063 } } peers { permanent_uuid: "bbdbef92d57b42cba823bbed27dfd341" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38901 } } peers { permanent_uuid: "de177e9eae86499eaf8fc793a688eed3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33033 } }
I20260929 02:14:24.097842 4781 raft_consensus.cc:374] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:14:24.098049 4781 raft_consensus.cc:729] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de177e9eae86499eaf8fc793a688eed3, State: Initialized, Role: FOLLOWER
I20260929 02:14:24.098142 4779 consensus_queue.cc:260] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6477a5027a9145b3bbcf4cab8b90e896" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38063 } } peers { permanent_uuid: "bbdbef92d57b42cba823bbed27dfd341" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38901 } } peers { permanent_uuid: "de177e9eae86499eaf8fc793a688eed3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33033 } }
I20260929 02:14:24.099184 4781 consensus_queue.cc:260] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6477a5027a9145b3bbcf4cab8b90e896" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38063 } } peers { permanent_uuid: "bbdbef92d57b42cba823bbed27dfd341" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38901 } } peers { permanent_uuid: "de177e9eae86499eaf8fc793a688eed3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33033 } }
I20260929 02:14:24.099704 4626 heartbeater.cc:503] Master 127.24.216.190:45139 was elected leader, sending a full tablet report...
I20260929 02:14:24.100677 4779 ts_tablet_manager.cc:1428] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896: Time spent starting tablet: real 0.018s user 0.013s sys 0.002s
I20260929 02:14:24.101969 4781 ts_tablet_manager.cc:1428] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260929 02:14:24.113279 4786 raft_consensus.cc:482] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:14:24.113552 4786 raft_consensus.cc:504] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6477a5027a9145b3bbcf4cab8b90e896" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38063 } } peers { permanent_uuid: "bbdbef92d57b42cba823bbed27dfd341" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38901 } } peers { permanent_uuid: "de177e9eae86499eaf8fc793a688eed3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33033 } }
I20260929 02:14:24.115474 4786 leader_election.cc:288] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbdbef92d57b42cba823bbed27dfd341 (127.24.216.131:38901), de177e9eae86499eaf8fc793a688eed3 (127.24.216.129:33033)
I20260929 02:14:24.124547 4776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.115835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:43605 (local address 127.24.216.131:38901)
0929 02:14:24.116017 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:14:24.116032 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:14:24.116596 (+ 564us) server_negotiation.cc:407] Connection header received
0929 02:14:24.116970 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:14:24.116977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:14:24.117063 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:14:24.117123 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:14:24.118557 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.119621 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.121997 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.123527 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.123747 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.123823 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 02:14:24.124067 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 02:14:24.124185 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:14:24.124627 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.115700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:38901 (local address 127.24.216.130:43605)
0929 02:14:24.116504 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 02:14:24.116522 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:14:24.116760 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:14:24.117372 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:14:24.117378 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:14:24.117404 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:14:24.118365 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:24.118379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.119752 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.119757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:24.121807 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:24.121818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.123747 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.123753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:24.123905 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.123928 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:14:24.124033 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 02:14:24.124164 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":21,"thread_start_us":481,"threads_started":1}
I20260929 02:14:24.124799 4778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.116793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:44011 (local address 127.24.216.129:33033)
0929 02:14:24.117067 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 02:14:24.117073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:14:24.117719 (+ 646us) server_negotiation.cc:407] Connection header received
0929 02:14:24.118456 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:14:24.118462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:14:24.118541 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:14:24.118623 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:14:24.119229 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.120596 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.123053 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.123888 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.124082 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.124148 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 02:14:24.124450 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 02:14:24.124567 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 02:14:24.125224 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.116570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:33033 (local address 127.24.216.130:44011)
0929 02:14:24.117602 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0929 02:14:24.117621 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:14:24.117857 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:14:24.118634 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:14:24.118638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:14:24.118659 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:14:24.119088 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:24.119096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.120754 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.120758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:24.122884 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:14:24.122899 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.124108 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.124114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:14:24.124305 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.124327 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:14:24.124430 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 02:14:24.124565 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":1,"thread_start_us":444,"threads_started":1}
I20260929 02:14:24.126077 4715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1ece05a53e483fb47dffc1468f9c0c" candidate_uuid: "6477a5027a9145b3bbcf4cab8b90e896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbdbef92d57b42cba823bbed27dfd341" is_pre_election: true
I20260929 02:14:24.126354 4438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1ece05a53e483fb47dffc1468f9c0c" candidate_uuid: "6477a5027a9145b3bbcf4cab8b90e896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de177e9eae86499eaf8fc793a688eed3" is_pre_election: true
I20260929 02:14:24.126555 4715 raft_consensus.cc:2454] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6477a5027a9145b3bbcf4cab8b90e896 in term 0.
I20260929 02:14:24.126859 4438 raft_consensus.cc:2454] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6477a5027a9145b3bbcf4cab8b90e896 in term 0.
I20260929 02:14:24.127205 4514 leader_election.cc:302] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6477a5027a9145b3bbcf4cab8b90e896, bbdbef92d57b42cba823bbed27dfd341; no voters:
I20260929 02:14:24.127594 4786 raft_consensus.cc:2781] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:14:24.127758 4786 raft_consensus.cc:482] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:14:24.127874 4786 raft_consensus.cc:3037] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:14:24.130115 4786 raft_consensus.cc:504] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6477a5027a9145b3bbcf4cab8b90e896" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38063 } } peers { permanent_uuid: "bbdbef92d57b42cba823bbed27dfd341" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38901 } } peers { permanent_uuid: "de177e9eae86499eaf8fc793a688eed3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33033 } }
I20260929 02:14:24.130908 4786 leader_election.cc:288] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [CANDIDATE]: Term 1 election: Requested vote from peers bbdbef92d57b42cba823bbed27dfd341 (127.24.216.131:38901), de177e9eae86499eaf8fc793a688eed3 (127.24.216.129:33033)
I20260929 02:14:24.131286 4715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1ece05a53e483fb47dffc1468f9c0c" candidate_uuid: "6477a5027a9145b3bbcf4cab8b90e896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbdbef92d57b42cba823bbed27dfd341"
I20260929 02:14:24.131376 4438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1ece05a53e483fb47dffc1468f9c0c" candidate_uuid: "6477a5027a9145b3bbcf4cab8b90e896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de177e9eae86499eaf8fc793a688eed3"
I20260929 02:14:24.131485 4715 raft_consensus.cc:3037] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:14:24.131608 4438 raft_consensus.cc:3037] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:14:24.133745 4438 raft_consensus.cc:2454] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6477a5027a9145b3bbcf4cab8b90e896 in term 1.
I20260929 02:14:24.133757 4715 raft_consensus.cc:2454] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6477a5027a9145b3bbcf4cab8b90e896 in term 1.
I20260929 02:14:24.134101 4514 leader_election.cc:302] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6477a5027a9145b3bbcf4cab8b90e896, de177e9eae86499eaf8fc793a688eed3; no voters:
I20260929 02:14:24.134491 4786 raft_consensus.cc:2781] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:14:24.135025 4786 raft_consensus.cc:686] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [term 1 LEADER]: Becoming Leader. State: Replica: 6477a5027a9145b3bbcf4cab8b90e896, State: Running, Role: LEADER
I20260929 02:14:24.135670 4786 consensus_queue.cc:237] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6477a5027a9145b3bbcf4cab8b90e896" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38063 } } peers { permanent_uuid: "bbdbef92d57b42cba823bbed27dfd341" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38901 } } peers { permanent_uuid: "de177e9eae86499eaf8fc793a688eed3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33033 } }
I20260929 02:14:24.140528 4269 catalog_manager.cc:5495] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6477a5027a9145b3bbcf4cab8b90e896 (127.24.216.130). New cstate: current_term: 1 leader_uuid: "6477a5027a9145b3bbcf4cab8b90e896" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6477a5027a9145b3bbcf4cab8b90e896" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbdbef92d57b42cba823bbed27dfd341" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de177e9eae86499eaf8fc793a688eed3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33033 } health_report { overall_health: UNKNOWN } } }
I20260929 02:14:24.174220 4774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.168738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45810 (local address 127.24.216.130:38063)
0929 02:14:24.168926 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:14:24.168934 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:14:24.169001 (+ 67us) server_negotiation.cc:407] Connection header received
0929 02:14:24.169251 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:14:24.169255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:14:24.169332 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:14:24.169519 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:14:24.169864 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.170679 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.172058 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.172448 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.172593 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.173065 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:14:24.173560 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:14:24.173762 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 02:14:24.173818 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 02:14:24.173897 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:14:24.182068 4715 raft_consensus.cc:1261] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341 [term 1 FOLLOWER]: Refusing update from remote peer 6477a5027a9145b3bbcf4cab8b90e896: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:14:24.182122 4438 raft_consensus.cc:1261] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3 [term 1 FOLLOWER]: Refusing update from remote peer 6477a5027a9145b3bbcf4cab8b90e896: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:14:24.183094 4790 consensus_queue.cc:1035] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbdbef92d57b42cba823bbed27dfd341" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:14:24.183422 4786 consensus_queue.cc:1035] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de177e9eae86499eaf8fc793a688eed3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:14:24.196416 4793 mvcc.cc:205] Tried to move back new op lower bound from 7334494470880284672 to 7334494470703960064. Current Snapshot: MvccSnapshot[applied={T|T < 7334494470880284672}]
I20260929 02:14:24.196764 4794 mvcc.cc:205] Tried to move back new op lower bound from 7334494470880284672 to 7334494470703960064. Current Snapshot: MvccSnapshot[applied={T|T < 7334494470880284672}]
I20260929 02:14:24.213604 4269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:14:24.213884 4269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:14:24.218320 4269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P faefc1e5eb9b421f8de1c444376b2473: Sending DeleteTablet for 3 replicas of tablet 9b1ece05a53e483fb47dffc1468f9c0c
I20260929 02:14:24.219587 4561 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b1ece05a53e483fb47dffc1468f9c0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:14:24 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:45800
I20260929 02:14:24.220536 4695 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b1ece05a53e483fb47dffc1468f9c0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:14:24 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:38474
I20260929 02:14:24.220608 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 4350
I20260929 02:14:24.221168 4418 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b1ece05a53e483fb47dffc1468f9c0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:14:24 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:35516
I20260929 02:14:24.223417 4804 tablet_replica.cc:330] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341: stopping tablet replica
I20260929 02:14:24.223920 4804 raft_consensus.cc:2229] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:14:24.224488 4804 raft_consensus.cc:2258] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:14:24.226186 4805 tablet_replica.cc:330] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3: stopping tablet replica
I20260929 02:14:24.226552 4805 raft_consensus.cc:2229] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:14:24.226981 4805 raft_consensus.cc:2258] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:14:24.226996 4804 ts_tablet_manager.cc:1905] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:14:24.231506 4805 ts_tablet_manager.cc:1905] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:14:24.236894 4804 ts_tablet_manager.cc:1918] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:14:24.237047 4804 log.cc:1196] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/9b1ece05a53e483fb47dffc1468f9c0c
I20260929 02:14:24.237565 4804 ts_tablet_manager.cc:1939] T 9b1ece05a53e483fb47dffc1468f9c0c P bbdbef92d57b42cba823bbed27dfd341: Deleting consensus metadata
I20260929 02:14:24.238392 4805 ts_tablet_manager.cc:1918] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:14:24.238561 4805 log.cc:1196] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/9b1ece05a53e483fb47dffc1468f9c0c
I20260929 02:14:24.239151 4805 ts_tablet_manager.cc:1939] T 9b1ece05a53e483fb47dffc1468f9c0c P de177e9eae86499eaf8fc793a688eed3: Deleting consensus metadata
I20260929 02:14:24.239591 4254 catalog_manager.cc:4841] TS bbdbef92d57b42cba823bbed27dfd341 (127.24.216.131:38901): tablet 9b1ece05a53e483fb47dffc1468f9c0c (table test-table [id=ad000645ad59469d8a7bec9c39ef6b5a]) successfully deleted
I20260929 02:14:24.240947 4254 catalog_manager.cc:4841] TS de177e9eae86499eaf8fc793a688eed3 (127.24.216.129:33033): tablet 9b1ece05a53e483fb47dffc1468f9c0c (table test-table [id=ad000645ad59469d8a7bec9c39ef6b5a]) successfully deleted
I20260929 02:14:24.241334 4803 tablet_replica.cc:330] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896: stopping tablet replica
I20260929 02:14:24.241807 4803 raft_consensus.cc:2229] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:14:24.242612 4803 raft_consensus.cc:2258] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 02:14:24 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:14:24.244510 4803 ts_tablet_manager.cc:1905] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:14:24.251783 4803 ts_tablet_manager.cc:1918] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:14:24.251996 4803 log.cc:1196] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/9b1ece05a53e483fb47dffc1468f9c0c
I20260929 02:14:24.252511 4803 ts_tablet_manager.cc:1939] T 9b1ece05a53e483fb47dffc1468f9c0c P 6477a5027a9145b3bbcf4cab8b90e896: Deleting consensus metadata
I20260929 02:14:24.252985 4778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.221915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.24.216.129:33033)
0929 02:14:24.222084 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:14:24.222090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:14:24.225484 (+ 3394us) server_negotiation.cc:407] Connection header received
0929 02:14:24.225650 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:14:24.225656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:14:24.225741 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:14:24.225848 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:14:24.231897 (+ 6049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.233106 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.236008 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.236734 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.236867 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.245457 (+ 8590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:14:24.245473 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:14:24.245489 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:14:24.245537 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:14:24.249422 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:24.250397 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:24.250406 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:24.250409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:24.250572 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:24.251014 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:24.251019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:24.251023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:24.251481 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:14:24.251797 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 02:14:24.252242 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 02:14:24.252520 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:14:24.253906 4478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:14:24.254231 4254 catalog_manager.cc:4841] TS 6477a5027a9145b3bbcf4cab8b90e896 (127.24.216.130:38063): tablet 9b1ece05a53e483fb47dffc1468f9c0c (table test-table [id=ad000645ad59469d8a7bec9c39ef6b5a]) successfully deleted
I20260929 02:14:24.345806 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 4350
I20260929 02:14:24.365046 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 4489
Sep 29 02:14:24 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:14:24.381187 4774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.365970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45816 (local address 127.24.216.130:38063)
0929 02:14:24.366143 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:14:24.366193 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0929 02:14:24.366212 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:14:24.366695 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:14:24.366700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:14:24.366776 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:14:24.366889 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:14:24.368141 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.368997 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.370355 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.370891 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.371023 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.375113 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:14:24.375123 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:14:24.375138 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:14:24.375167 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:14:24.378225 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:24.378869 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:24.378882 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:24.378884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:24.379113 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:24.379529 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:24.379533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:24.379535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:24.380031 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:14:24.380365 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 02:14:24.380619 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 02:14:24.380838 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:14:24.381947 4621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:14:24.495769 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 4489
I20260929 02:14:24.511202 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 4632
Sep 29 02:14:24 dist-test-slave-9k0h krb5kdc[3794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:14:24.526378 4776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:14:24.512026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.24.216.131:38901)
0929 02:14:24.512234 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:14:24.512240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:14:24.512260 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:14:24.512319 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:14:24.512323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:14:24.512391 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:14:24.512474 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:14:24.513726 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.514532 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.516262 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:14:24.516749 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:14:24.516857 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:14:24.521206 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:14:24.521218 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:14:24.521234 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:14:24.521264 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:14:24.523868 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:24.524482 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:24.524489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:24.524492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:24.524599 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:14:24.525037 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:14:24.525041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:14:24.525043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:14:24.525414 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:14:24.525625 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 02:14:24.525881 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 02:14:24.526037 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 02:14:24.526991 4755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:14:24.615993 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 4632
I20260929 02:14:24.633045 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 4238
I20260929 02:14:24.634207 4300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:14:24.746065 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 4238
I20260929 02:14:24.769608 25442 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 02:14:25.104437 25442 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 02:14:25.104655 25442 mini_ranger.cc:66] Stopping Ranger...
I20260929 02:14:25.450196 25442 mini_ranger.cc:68] Stopped Ranger
2026-09-29 02:14:25.450 UTC [3824] LOG: received smart shutdown request
2026-09-29 02:14:25.453 UTC [3824] LOG: background worker "logical replication launcher" (PID 3833) exited with exit code 1
2026-09-29 02:14:25.454 UTC [3828] LOG: shutting down
2026-09-29 02:14:25.608 UTC [3824] LOG: database system is shut down
2026-09-29T02:14:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47778 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:14:25 dist-test-slave-9k0h krb5kdc[4819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:14:25 dist-test-slave-9k0h krb5kdc[4819](info): set up 2 sockets
Sep 29 02:14:25 dist-test-slave-9k0h krb5kdc[4819](info): commencing operation
krb5kdc: starting...
W20260929 02:14:26.878571 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:14:27 dist-test-slave-9k0h krb5kdc[4819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:14:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:14:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.216.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.216.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/rangeradmin_127.24.216.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/rangeradmin_127.24.216.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.216.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.216.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/rangerlookup_127.24.216.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/rangerlookup_127.24.216.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.216.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.216.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/HTTP_127.24.216.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/HTTP_127.24.216.148@KRBTEST.COM.keytab.
I20260929 02:14:27.193018 25442 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGNPMkl/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/postgres -l logfile start
2026-09-29 02:14:29.731 UTC [4849] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 02:14:29.732 UTC [4849] LOG: listening on IPv4 address "127.24.216.148", port 60049
2026-09-29 02:14:29.735 UTC [4849] LOG: listening on Unix socket "/tmp/.s.PGSQL.60049"
2026-09-29 02:14:29.757 UTC [4852] LOG: database system was shut down at 2026-09-29 02:14:28 UTC
2026-09-29 02:14:29.760 UTC [4849] LOG: database system is ready to accept connections
I20260929 02:14:30.778331 25442 mini_postgres.cc:96] Postgres bound to 60049
2026-09-29 02:14:30.796 UTC [4860] FATAL: database "slave" does not exist
127.24.216.148:60049 - accepting connections
I20260929 02:14:30.796873 25442 mini_ranger.cc:161] Starting Ranger...
I20260929 02:14:30.822659 25442 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 02:14:31.318307 25442 mini_ranger.cc:87] Created ranger Postgres database
I20260929 02:14:31.318398 25442 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-admin
2026-09-29 02:14:31,584 [I] DB FLAVOR :POSTGRES
2026-09-29 02:14:31,585 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:14:31,585 [I] Checking connection..
2026-09-29 02:14:31,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 02:14:31,844 [I] Checking connection passed.
2026-09-29 02:14:31,845 [I] --------- Verifying version history table ---------
2026-09-29 02:14:31,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:14:32,124 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:14:32,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:14:32,373 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:14:32,373 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 02:14:32,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 02:14:32.611 UTC [4924] WARNING: there is no transaction in progress
2026-09-29 02:14:32,617 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 02:14:32,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:14:32,860 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 02:14:32,860 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:14:32,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 02:14:33,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 02:14:33,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9k0h.c.gcp-upstream.internal','N') ;"
2026-09-29 02:14:33,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:14:33,873 [I] Table x_portal_user does not exist in database ranger
2026-09-29 02:14:33,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:14:34,102 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 02:14:34,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:14:34,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:14:34,555 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 02:14:34,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 02:14:35.518 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.535 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.550 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.566 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.581 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.599 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.651 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.658 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.671 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.681 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.691 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.705 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.714 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.721 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.728 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:35.734 UTC [5032] WARNING: there is no transaction in progress
2026-09-29 02:14:36,368 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 02:14:36,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 02:14:36,601 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 02:14:36,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:14:36,831 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 02:14:36,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:14:37,091 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 02:14:37,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:14:37,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:14:37,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9k0h.c.gcp-upstream.internal';"
2026-09-29 02:14:37,821 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 02:14:37,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 02:14:38,074 [I] DB_PATCHES have already been applied
I20260929 02:14:38.080622 25442 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGNPMkl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 02:14:38.080704 25442 mini_ranger.cc:193] Using host: 127.24.216.148
I20260929 02:14:38.096669 25442 mini_ranger.cc:227] Ranger admin URL: http://127.24.216.148:53003
Sep 29, 2026 2:14:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:14:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:14:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:14:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:14:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:14:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:14:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 02:14:39.216267 25442 mini_ranger.cc:160] Time spent starting Ranger: real 8.419s user 0.007s sys 0.035s
Sep 29, 2026 2:14:59 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:15:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 2:15:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:15:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:15:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:15:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:15:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:15:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:15:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:15:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:15:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:15:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:15:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:15:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:15:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:15:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:15:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:15:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 02:15:01.667389 25442 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.216.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.216.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/rangerkms_127.24.216.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.216.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/rangerkms_127.24.216.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.216.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.216.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.216.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/HTTP_127.24.216.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.216.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/HTTP_127.24.216.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 02:15:01 dist-test-slave-9k0h krb5kdc[4819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648101, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 02:15:01.868924 25442 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 02:15:01.892643 25442 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 02:15:02.912643 25442 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 02:15:02.912765 25442 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-kms
I20260929 02:15:02.912811 25442 mini_ranger_kms.cc:227] Using postgres at 127.24.216.148:60049
2026-09-29 02:15:03,293 [I] DB FLAVOR :POSTGRES
2026-09-29 02:15:03,293 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:15:03,293 [I] Checking connection
2026-09-29 02:15:03,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 02:15:03,552 [I] connection success
2026-09-29 02:15:03,553 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 02:15:03,553 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 02:15:03,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 02:15:03,823 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 02:15:03,824 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:15:03,824 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 02:15:03,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.216.148:60049/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 02:15:04,164 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 02:15:04.627975 25442 mini_ranger_kms.cc:310] Created kms service
I20260929 02:15:04.822316 25442 mini_ranger_kms.cc:326] Created kudu user
I20260929 02:15:04.866065 25442 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 02:15:04 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648101, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
I20260929 02:15:05.283907 25442 mini_ranger_kms.cc:384] Added ranger policy
I20260929 02:15:05.284235 25442 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGNPMkl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGNPMkl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 02:15:05.284318 25442 mini_ranger_kms.cc:240] Using host: 127.24.216.148
I20260929 02:15:05.302464 25442 mini_ranger_kms.cc:276] Ranger KMS PID: 5213
I20260929 02:15:05.302606 25442 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.216.148:49043
Sep 29, 2026 2:15:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:15:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:15:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:15:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:15:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:15:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 02:15:07.774833 25442 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.906s user 0.069s sys 0.333s
I20260929 02:15:07.775089 25442 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 02:15:07.775261 25442 mini_ranger_kms.cc:401] 127.24.216.148:49043/kms/v1/keys
Sep 29 02:15:08 dist-test-slave-9k0h krb5kdc[4819](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790648108, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.216.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 02:15:08 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648108, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.216.148@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
Sep 29 02:15:08 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648108, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.216.148@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
Sep 29 02:15:11 dist-test-slave-9k0h krb5kdc[4819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:15:11.319096 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:36699
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:36699
--ranger_config_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.216.148:49043/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:11.669046 5263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:11.669363 5263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:11.669520 5263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:11.676869 5263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:15:11.676982 5263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:11.677083 5263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:11.677192 5263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:15:11.677295 5263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:15:11.687829 5263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.216.148:49043/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:36699
--ranger_config_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:36699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:11.689286 5263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:11.690833 5263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:11.705008 5269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:11.705129 5263 server_base.cc:1034] running on GCE node
W20260929 02:15:11.705066 5271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:11.705008 5268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:11.706308 5263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:11.707770 5263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:11.708976 5263 hybrid_clock.cc:648] HybridClock initialized: now 1790648111708936 us; error 38 us; skew 500 ppm
I20260929 02:15:11.709439 5263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:11 dist-test-slave-9k0h krb5kdc[4819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648111, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:11.718518 5263 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:15:11.720480 5263 webserver.cc:460] Webserver started at http://127.24.216.190:34581/ using document root <none> and password file <none>
I20260929 02:15:11.721179 5263 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:11.721284 5263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:11.721566 5263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:15:11 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648111, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
I20260929 02:15:11.781818 5263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb"
format_stamp: "Formatted at 2026-09-29 02:15:11 on dist-test-slave-9k0h"
server_key: "b016686fb47d43f369e2b43c529ff831"
server_key_iv: "389922ca5cc04e1ac39d61fbd8e14b85"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:11.783015 5263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb"
format_stamp: "Formatted at 2026-09-29 02:15:11 on dist-test-slave-9k0h"
server_key: "b016686fb47d43f369e2b43c529ff831"
server_key_iv: "389922ca5cc04e1ac39d61fbd8e14b85"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:11.790218 5263 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260929 02:15:11.809278 5278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:11.811390 5263 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:15:11.811650 5263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb"
format_stamp: "Formatted at 2026-09-29 02:15:11 on dist-test-slave-9k0h"
server_key: "b016686fb47d43f369e2b43c529ff831"
server_key_iv: "389922ca5cc04e1ac39d61fbd8e14b85"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:11.812000 5263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:11.836313 5263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:11.840732 5263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:11.841077 5263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:11.861570 5263 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:36699
I20260929 02:15:11.861569 5329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:36699 every 8 connection(s)
I20260929 02:15:11.863132 5263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:15:11.868000 5330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:11.870056 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 5263
I20260929 02:15:11.870334 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:15:11.870994 25442 external_mini_cluster.cc:1428] Setting key 9a3c42459e5769d943c89e1678b5d21b
I20260929 02:15:11.882750 5330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb: Bootstrap starting.
I20260929 02:15:11.887055 5330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:11.888621 5330 log.cc:824] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:11.892303 5330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb: No bootstrap required, opened a new log
Sep 29 02:15:11 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:11.903218 5330 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36699 } }
I20260929 02:15:11.903735 5330 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:11.903889 5330 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc3d8681bfeb4abab7ccf5f8da465dfb, State: Initialized, Role: FOLLOWER
I20260929 02:15:11.904630 5330 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36699 } }
I20260929 02:15:11.904913 5330 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:15:11.905092 5330 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:15:11.905310 5330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:11.908692 5330 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36699 } }
I20260929 02:15:11.909317 5330 leader_election.cc:302] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc3d8681bfeb4abab7ccf5f8da465dfb; no voters:
I20260929 02:15:11.909973 5333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:11.873313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.24.216.190:36699)
0929 02:15:11.887827 (+ 14514us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:11.887841 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:11.887883 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:15:11.890305 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:11.890334 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:11.890739 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:11.891134 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:11.892864 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:11.894531 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:11.895921 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:11.896676 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:11.896904 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:11.901619 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:11.901630 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:11.901648 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:11.901687 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:11.905304 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:11.906075 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:11.906085 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:11.906095 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:11.906335 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:11.906756 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:11.906763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:11.906766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:11.907414 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:11.907738 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:11.908225 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 02:15:11.908689 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14209,"thread_start_us":15398,"threads_started":1}
I20260929 02:15:11.910383 5330 leader_election.cc:288] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:15:11.910665 5335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:15:11.914701 5335 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 1 LEADER]: Becoming Leader. State: Replica: fc3d8681bfeb4abab7ccf5f8da465dfb, State: Running, Role: LEADER
I20260929 02:15:11.915421 5335 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36699 } }
I20260929 02:15:11.917114 5330 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:15:11.918242 5336 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc3d8681bfeb4abab7ccf5f8da465dfb. Latest consensus state: current_term: 1 leader_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36699 } } }
I20260929 02:15:11.918628 5336 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:11.919171 5337 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36699 } } }
I20260929 02:15:11.919443 5337 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:11.931233 5340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:15:11.936969 5340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:15:11.953411 5340 catalog_manager.cc:1347] Generated new cluster ID: 8f9f9cda2afb4fe6bd5cf586474bcb61
I20260929 02:15:11.953667 5340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:15:11.988785 5340 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:15:11.990902 5340 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:15:12.003476 5340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb: Generated new TSK 0
I20260929 02:15:12.004733 5340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:15:12.008083 5330 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:15:12.822974 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.216.148:49043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36699
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:15:13 dist-test-slave-9k0h krb5kdc[4819](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790648113, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 02:15:13.257750 5376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:13.258100 5376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:13.258298 5376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:13.267814 5376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:13.267974 5376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:13.268198 5376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:15:13.280326 5376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.216.148:49043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:13.282121 5376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:13.284010 5376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:13.309988 5386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:13.315491 5387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:13.316195 5389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:13.317051 5376 server_base.cc:1034] running on GCE node
I20260929 02:15:13.317878 5376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:13.319918 5376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:13.321280 5376 hybrid_clock.cc:648] HybridClock initialized: now 1790648113321169 us; error 92 us; skew 500 ppm
I20260929 02:15:13.321667 5376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:13 dist-test-slave-9k0h krb5kdc[4819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648113, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:13.325311 5376 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:15:13.330555 5376 webserver.cc:460] Webserver started at http://127.24.216.129:41233/ using document root <none> and password file <none>
I20260929 02:15:13.331367 5376 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:13.331518 5376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:13.331874 5376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:15:13 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648113, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
I20260929 02:15:13.352550 5376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "9bf396a65a2b41eaadc27427f0de83bc"
format_stamp: "Formatted at 2026-09-29 02:15:13 on dist-test-slave-9k0h"
server_key: "8359007d062fceb1a3245ed2653c96a0"
server_key_iv: "8cd89cfb93da0d69cc3982fee0f2a776"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:13.353672 5376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "9bf396a65a2b41eaadc27427f0de83bc"
format_stamp: "Formatted at 2026-09-29 02:15:13 on dist-test-slave-9k0h"
server_key: "8359007d062fceb1a3245ed2653c96a0"
server_key_iv: "8cd89cfb93da0d69cc3982fee0f2a776"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:13.376827 5376 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.012s sys 0.012s
I20260929 02:15:13.409507 5396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:13.411576 5376 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:15:13.411783 5376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "9bf396a65a2b41eaadc27427f0de83bc"
format_stamp: "Formatted at 2026-09-29 02:15:13 on dist-test-slave-9k0h"
server_key: "8359007d062fceb1a3245ed2653c96a0"
server_key_iv: "8cd89cfb93da0d69cc3982fee0f2a776"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:13.412066 5376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:13.440528 5376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:13.446691 5376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:13.447125 5376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:13.454749 5376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:13.463373 5376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:13.463501 5376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:13.463634 5376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:13.463701 5376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:13.556247 5376 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:34191
I20260929 02:15:13.556274 5508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:34191 every 8 connection(s)
I20260929 02:15:13.558009 5376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:15:13.558949 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 5376
I20260929 02:15:13.559161 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:15:13.559991 25442 external_mini_cluster.cc:1428] Setting key a9732a572c05e49b890e74f84f16bc8a
Sep 29 02:15:13 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648113, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:13.608858 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:13.569571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36699 (local address 127.24.216.129:54801)
0929 02:15:13.583384 (+ 13813us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:13.583443 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:13.585293 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:13.586342 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:13.586361 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:13.587249 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:13.588338 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:13.588366 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:13.590257 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:13.590263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:13.592384 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:13.592399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:13.593767 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:13.593776 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:13.594109 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:13.594957 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:13.594994 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:13.598361 (+ 3367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:13.603319 (+ 4958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:13.603332 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:13.603358 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:13.604092 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:13.604576 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:13.604596 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:13.604602 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:13.604874 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:13.605683 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:13.605694 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:13.606261 (+ 567us) client_negotiation.cc:769] Sending connection context
0929 02:15:13.606609 (+ 348us) client_negotiation.cc:240] Negotiation successful
0929 02:15:13.607264 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13489,"thread_start_us":13247,"threads_started":1}
I20260929 02:15:13.610193 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:13.570279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:54801 (local address 127.24.216.190:36699)
0929 02:15:13.582721 (+ 12442us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:13.582732 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:13.585691 (+ 2959us) server_negotiation.cc:407] Connection header received
0929 02:15:13.585752 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:13.585755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:13.585825 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:13.585964 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:13.588584 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:13.590028 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:13.592588 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:13.593300 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:13.593438 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:13.598684 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:13.598694 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:13.598702 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:13.598759 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:13.603110 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:13.604256 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:13.604265 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:13.604272 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:13.604436 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:13.605022 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:13.605029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:13.605035 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:13.605435 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:13.605606 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:13.609436 (+ 3830us) server_negotiation.cc:299] Negotiation successful
0929 02:15:13.609717 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12223,"thread_start_us":12075,"threads_started":1}
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
I20260929 02:15:13.612676 5509 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36699
I20260929 02:15:13.613001 5509 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:13.613878 5509 heartbeater.cc:511] Master 127.24.216.190:36699 requested a full tablet report, sending...
I20260929 02:15:13.616770 5295 ts_manager.cc:194] Registered new tserver with Master: 9bf396a65a2b41eaadc27427f0de83bc (127.24.216.129:34191)
I20260929 02:15:13.619524 5295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:54801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Sep 29 02:15:13 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648113, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
W20260929 02:15:13.669843 25442 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.24.216.130: real 0.106s user 0.000s sys 0.021s
I20260929 02:15:13.670871 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.216.148:49043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36699
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:15:13 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648113, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
W20260929 02:15:14.179193 5515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:14.179740 5515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:14.179939 5515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:14.191733 5515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:14.191978 5515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:14.192225 5515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:15:14.208300 5515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.216.148:49043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:14.210393 5515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:14.212306 5515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:14.242231 5522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:14.242607 5523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:14.246848 5515 server_base.cc:1034] running on GCE node
W20260929 02:15:14.247269 5525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:14.248117 5515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:14.249478 5515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:14.250788 5515 hybrid_clock.cc:648] HybridClock initialized: now 1790648114250727 us; error 55 us; skew 500 ppm
I20260929 02:15:14.251225 5515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:14 dist-test-slave-9k0h krb5kdc[4819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648114, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:14.256032 5515 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:15:14.258746 5515 webserver.cc:460] Webserver started at http://127.24.216.130:39515/ using document root <none> and password file <none>
I20260929 02:15:14.260066 5515 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:14.260259 5515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:14.260728 5515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:15:14 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648114, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
I20260929 02:15:14.287119 5515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8"
format_stamp: "Formatted at 2026-09-29 02:15:14 on dist-test-slave-9k0h"
server_key: "82b0c2ba2e41ac9f63595853eab65c2c"
server_key_iv: "0f01829c2c981a7a30a044f7f709170d"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:14.288414 5515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8"
format_stamp: "Formatted at 2026-09-29 02:15:14 on dist-test-slave-9k0h"
server_key: "82b0c2ba2e41ac9f63595853eab65c2c"
server_key_iv: "0f01829c2c981a7a30a044f7f709170d"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:14.295075 5515 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 02:15:14.315198 5539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:14.317641 5515 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:15:14.317843 5515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8"
format_stamp: "Formatted at 2026-09-29 02:15:14 on dist-test-slave-9k0h"
server_key: "82b0c2ba2e41ac9f63595853eab65c2c"
server_key_iv: "0f01829c2c981a7a30a044f7f709170d"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:14.318233 5515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:14.348924 5515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:14.354692 5515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:14.355046 5515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:14.356312 5515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:14.358191 5515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:14.358283 5515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 02:15:14.358455 5515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:14.358520 5515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:14.410872 5515 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:40975
I20260929 02:15:14.410914 5651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:40975 every 8 connection(s)
I20260929 02:15:14.412736 5515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:15:14.422915 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 5515
I20260929 02:15:14.423120 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:15:14.423727 25442 external_mini_cluster.cc:1428] Setting key a89ae890046b86b549737279c09c7606
Sep 29 02:15:14 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648114, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:14.448891 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:14.417108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:38319 (local address 127.24.216.190:36699)
0929 02:15:14.422712 (+ 5604us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:14.422721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:14.426200 (+ 3479us) server_negotiation.cc:407] Connection header received
0929 02:15:14.426272 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:14.426277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:14.426372 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:14.426467 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:14.428994 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:14.430276 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:14.433064 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:14.433812 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:14.433934 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:14.440162 (+ 6228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:14.440172 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:14.440180 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:14.440231 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:14.444318 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:14.445508 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:14.445517 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:14.445523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:14.445697 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:14.446325 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:14.446333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:14.446339 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:14.446754 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:14.446879 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:14.448050 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0929 02:15:14.448355 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5424,"thread_start_us":5295,"threads_started":1}
I20260929 02:15:14.451962 5654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:14.416495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36699 (local address 127.24.216.130:38319)
0929 02:15:14.423379 (+ 6884us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:14.423456 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:14.425888 (+ 2432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:14.427072 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:14.427088 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:14.427820 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:14.428759 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:14.428787 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:14.430505 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:14.430514 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:14.432865 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:14.432884 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:14.435201 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:14.435211 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:14.435589 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:14.436456 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:14.436492 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:14.439805 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:14.444569 (+ 4764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:14.444595 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:14.444620 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:14.445340 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:14.445856 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:14.445863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:14.445869 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:14.446188 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:14.446996 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:14.447008 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:14.447533 (+ 525us) client_negotiation.cc:769] Sending connection context
0929 02:15:14.449512 (+ 1979us) client_negotiation.cc:240] Negotiation successful
0929 02:15:14.450235 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6531,"thread_start_us":6276,"threads_started":1}
I20260929 02:15:14.454200 5652 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36699
I20260929 02:15:14.454566 5652 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
I20260929 02:15:14.455479 5652 heartbeater.cc:511] Master 127.24.216.190:36699 requested a full tablet report, sending...
I20260929 02:15:14.457065 5295 ts_manager.cc:194] Registered new tserver with Master: df1d6d5f9096496c85b2d2a4e0bb1ed8 (127.24.216.130:40975)
I20260929 02:15:14.457993 5295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:38319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:15:14.490712 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.216.148:49043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36699
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:15:14.624070 5509 heartbeater.cc:503] Master 127.24.216.190:36699 was elected leader, sending a full tablet report...
W20260929 02:15:14.874897 5658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:14.875269 5658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:14.875445 5658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:14.883831 5658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:14.883967 5658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:14.884218 5658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:15:14.894805 5658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.216.148:49043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:14.896569 5658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:14.898392 5658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:14.912925 5663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:14.912952 5664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:14.912986 5666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:14.914430 5658 server_base.cc:1034] running on GCE node
I20260929 02:15:14.915118 5658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:14.916448 5658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:14.917802 5658 hybrid_clock.cc:648] HybridClock initialized: now 1790648114917723 us; error 57 us; skew 500 ppm
I20260929 02:15:14.918283 5658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:14 dist-test-slave-9k0h krb5kdc[4819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648114, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:14.921907 5658 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:15:14.923650 5658 webserver.cc:460] Webserver started at http://127.24.216.131:32973/ using document root <none> and password file <none>
I20260929 02:15:14.924335 5658 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:14.924500 5658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:14.924878 5658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:15:14 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648114, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
I20260929 02:15:14.942905 5658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "ba91392c21a3453fafc548fda59e9d4e"
format_stamp: "Formatted at 2026-09-29 02:15:14 on dist-test-slave-9k0h"
server_key: "27c390719ba1e1b3b370b072651025ce"
server_key_iv: "28af955a4e990556b868bcfb01ac5651"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:14.944025 5658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "ba91392c21a3453fafc548fda59e9d4e"
format_stamp: "Formatted at 2026-09-29 02:15:14 on dist-test-slave-9k0h"
server_key: "27c390719ba1e1b3b370b072651025ce"
server_key_iv: "28af955a4e990556b868bcfb01ac5651"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:14.951658 5658 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 02:15:14.967027 5673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:14.968642 5658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:15:14.968835 5658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "ba91392c21a3453fafc548fda59e9d4e"
format_stamp: "Formatted at 2026-09-29 02:15:14 on dist-test-slave-9k0h"
server_key: "27c390719ba1e1b3b370b072651025ce"
server_key_iv: "28af955a4e990556b868bcfb01ac5651"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:14.969158 5658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:14.989356 5658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:14.993402 5658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:14.993736 5658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:14.994835 5658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:14.996327 5658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:14.996400 5658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:14.996553 5658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:14.996611 5658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:15.033191 5658 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:36315
I20260929 02:15:15.033229 5785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:36315 every 8 connection(s)
I20260929 02:15:15.034922 5658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:15:15.035507 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 5658
I20260929 02:15:15.035718 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:15:15.036337 25442 external_mini_cluster.cc:1428] Setting key 0de9ba5bb18bcb99995a9a584f3a0fe4
Sep 29 02:15:15 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648114, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:15.057215 5788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:15.037752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:59373 (local address 127.24.216.190:36699)
0929 02:15:15.038382 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:15.038386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:15.039694 (+ 1308us) server_negotiation.cc:407] Connection header received
0929 02:15:15.041545 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:15.041549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:15.041626 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:15.041720 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:15.043367 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:15.044099 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:15.045876 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:15.046515 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:15.046627 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:15.050470 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:15.050476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:15.050479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:15.050513 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:15.053972 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:15.054885 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:15.054890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:15.054893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:15.055039 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:15.055530 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:15.055535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:15.055538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:15.055848 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:15.055978 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:15.056757 (+ 779us) server_negotiation.cc:299] Negotiation successful
0929 02:15:15.056945 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":857,"threads_started":1}
I20260929 02:15:15.058300 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:15.038571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36699 (local address 127.24.216.131:59373)
0929 02:15:15.039505 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:15.039559 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:15.041290 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:15.041979 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:15.041987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:15.042502 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:15.043192 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:15.043215 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:15.044295 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:15.044300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:15.045697 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:15.045711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:15.046755 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:15.046760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:15.047003 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:15.047944 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:15.047971 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:15.050193 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:15.054132 (+ 3939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:15.054139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:15.054222 (+ 83us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:15.054723 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:15.055159 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:15.055163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:15.055165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:15.055371 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:15.056025 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:15.056032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:15.056378 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 02:15:15.056620 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 02:15:15.057040 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":413,"threads_started":1}
I20260929 02:15:15.059825 5786 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36699
I20260929 02:15:15.060146 5786 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:15.060907 5786 heartbeater.cc:511] Master 127.24.216.190:36699 requested a full tablet report, sending...
I20260929 02:15:15.062484 5295 ts_manager.cc:194] Registered new tserver with Master: ba91392c21a3453fafc548fda59e9d4e (127.24.216.131:36315)
I20260929 02:15:15.063309 5295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:59373
I20260929 02:15:15.065620 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:15:15.067660 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 5376
Sep 29 02:15:15 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:15:15.085914 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:15.068574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.24.216.129:34191)
0929 02:15:15.069192 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:15.069202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:15.069261 (+ 59us) server_negotiation.cc:407] Connection header received
0929 02:15:15.069341 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:15.069348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:15.069546 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:15.069726 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:15.071291 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:15.072305 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:15.074008 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:15.074792 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:15.074983 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:15.079680 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:15.079693 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:15.079709 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:15.079749 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:15.082778 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:15.083552 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:15.083559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:15.083562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:15.083706 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:15.084131 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:15.084136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:15.084139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:15.084520 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:15.084926 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:15.085265 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 02:15:15.085537 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":275,"threads_started":1}
I20260929 02:15:15.087391 5504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:15.173653 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 5376
I20260929 02:15:15.191308 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 5515
Sep 29 02:15:15 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:15:15.214434 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:15.192293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.24.216.130:40975)
0929 02:15:15.192945 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:15.192953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:15.192979 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:15:15.193043 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:15.193050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:15.193279 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:15.193418 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:15.195060 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:15.196104 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:15.197603 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:15.198298 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:15.198415 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:15.208320 (+ 9905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:15.208337 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:15.208355 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:15.208408 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:15.211535 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:15.212356 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:15.212363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:15.212366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:15.212500 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:15.212842 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:15.212845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:15.212847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:15.213229 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:15.213473 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:15.213735 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 02:15:15.213960 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":371,"threads_started":1}
I20260929 02:15:15.216223 5647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:15.307574 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 5515
I20260929 02:15:15.324090 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 5658
Sep 29 02:15:15 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:15:15.341845 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:15.325025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.24.216.131:36315)
0929 02:15:15.325667 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:15.325674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:15.325703 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:15:15.325764 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:15.325770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:15.325998 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:15.326141 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:15.327531 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:15.328632 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:15.329974 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:15.330751 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:15.330868 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:15.335538 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:15.335553 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:15.335571 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:15.335615 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:15.339068 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:15.339891 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:15.339900 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:15.339902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:15.340067 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:15.340456 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:15.340460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:15.340462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:15.340833 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:15.341133 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:15.341350 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 02:15:15.341556 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":325,"threads_started":1}
I20260929 02:15:15.343228 5781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:15.427079 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 5658
I20260929 02:15:15.443362 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 5263
I20260929 02:15:15.444761 5325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:15.557737 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 5263
I20260929 02:15:15.583902 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:36699
--webserver_interface=127.24.216.190
--webserver_port=34581
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:36699
--ranger_config_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.216.148:49043/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.216.148:49043/kms with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:15.935643 5798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:15.935976 5798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:15.936195 5798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:15.944126 5798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:15:15.944257 5798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:15.944341 5798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:15.944453 5798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:15:15.944526 5798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:15:15.955422 5798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.216.148:49043/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:36699
--ranger_config_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:36699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=34581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:15.956809 5798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:15.958289 5798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:15.970638 5803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:15.970543 5806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:15.970583 5804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:15.971951 5798 server_base.cc:1034] running on GCE node
I20260929 02:15:15.972867 5798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:15.974442 5798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:15.975682 5798 hybrid_clock.cc:648] HybridClock initialized: now 1790648115975618 us; error 62 us; skew 500 ppm
I20260929 02:15:15.976065 5798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:15 dist-test-slave-9k0h krb5kdc[4819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648115, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:15.979315 5798 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:15:15.980940 5798 webserver.cc:460] Webserver started at http://127.24.216.190:34581/ using document root <none> and password file <none>
I20260929 02:15:15.981639 5798 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:15.981777 5798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:15:16 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648115, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
I20260929 02:15:16.051148 5798 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 02:15:16.054559 5814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:16.056233 5798 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:15:16.056455 5798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb"
format_stamp: "Formatted at 2026-09-29 02:15:11 on dist-test-slave-9k0h"
server_key: "b016686fb47d43f369e2b43c529ff831"
server_key_iv: "389922ca5cc04e1ac39d61fbd8e14b85"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:16.057472 5798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:16.088547 5798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:16.092907 5798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:16.093222 5798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:16.112703 5798 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:36699
I20260929 02:15:16.112725 5865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:36699 every 8 connection(s)
I20260929 02:15:16.114357 5798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:15:16.121610 5866 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:15:16.122239 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 5798
I20260929 02:15:16.123311 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.216.148:49043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:34191
--local_ip_for_outbound_sockets=127.24.216.129
--tserver_master_addrs=127.24.216.190:36699
--webserver_port=41233
--webserver_interface=127.24.216.129
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.216.148:49043/kms with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:15:16.127004 5866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb: Bootstrap starting.
I20260929 02:15:16.147720 5866 log.cc:824] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:16.157928 5866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:15:16.158568 5866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb: Bootstrap complete.
I20260929 02:15:16.168870 5866 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36699 } }
I20260929 02:15:16.169860 5866 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc3d8681bfeb4abab7ccf5f8da465dfb, State: Initialized, Role: FOLLOWER
I20260929 02:15:16.170728 5866 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36699 } }
I20260929 02:15:16.170995 5866 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:15:16.171159 5866 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:15:16.171355 5866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:15:16.175559 5866 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36699 } }
I20260929 02:15:16.176110 5866 leader_election.cc:302] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc3d8681bfeb4abab7ccf5f8da465dfb; no voters:
I20260929 02:15:16.176785 5866 leader_election.cc:288] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:15:16.176935 5870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:15:16.178319 5870 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [term 2 LEADER]: Becoming Leader. State: Replica: fc3d8681bfeb4abab7ccf5f8da465dfb, State: Running, Role: LEADER
I20260929 02:15:16.179030 5870 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36699 } }
I20260929 02:15:16.179378 5866 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:15:16.182314 5871 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36699 } } }
I20260929 02:15:16.182693 5871 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:16.181934 5872 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc3d8681bfeb4abab7ccf5f8da465dfb. Latest consensus state: current_term: 2 leader_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3d8681bfeb4abab7ccf5f8da465dfb" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36699 } } }
I20260929 02:15:16.183346 5872 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:16.191762 5876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:15:16.199172 5876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:15:16.201185 5876 catalog_manager.cc:1259] Loaded cluster ID: 8f9f9cda2afb4fe6bd5cf586474bcb61
I20260929 02:15:16.201399 5876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:15:16.206892 5876 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:15:16.208920 5876 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb: Loaded TSK: 0
I20260929 02:15:16.211014 5876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:15:16.216341 5866 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 02:15:16.593407 5868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:16.593962 5868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:16.594197 5868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:16.606367 5868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:16.606627 5868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:16.606832 5868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:15:16.623183 5868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.216.148:49043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:34191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=41233
--tserver_master_addrs=127.24.216.190:36699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:16.625120 5868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:16.626972 5868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:16.662897 5903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:16.665442 5904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:16.682739 5868 server_base.cc:1034] running on GCE node
W20260929 02:15:16.683283 5907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:16.684763 5868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:16.693768 5868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:16.695355 5868 hybrid_clock.cc:648] HybridClock initialized: now 1790648116695247 us; error 121 us; skew 500 ppm
I20260929 02:15:16.695851 5868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:16 dist-test-slave-9k0h krb5kdc[4819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648116, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:16.700304 5868 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:15:16.711992 5868 webserver.cc:460] Webserver started at http://127.24.216.129:41233/ using document root <none> and password file <none>
I20260929 02:15:16.713172 5868 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:16.713369 5868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:15:16 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648116, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
I20260929 02:15:16.764438 5868 fs_manager.cc:709] Time spent opening directory manager: real 0.028s user 0.011s sys 0.016s
I20260929 02:15:16.776598 5915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:16.778887 5868 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.008s
I20260929 02:15:16.779218 5868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "9bf396a65a2b41eaadc27427f0de83bc"
format_stamp: "Formatted at 2026-09-29 02:15:13 on dist-test-slave-9k0h"
server_key: "8359007d062fceb1a3245ed2653c96a0"
server_key_iv: "8cd89cfb93da0d69cc3982fee0f2a776"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:16.780854 5868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:16.817142 5868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:16.821532 5868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:16.821899 5868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:16.823167 5868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:16.824757 5868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:16.824846 5868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:16.824976 5868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:16.825089 5868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:16.875174 5868 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:34191
I20260929 02:15:16.875211 6027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:34191 every 8 connection(s)
I20260929 02:15:16.876701 5868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:15:16.885092 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 5868
I20260929 02:15:16.885957 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.216.148:49043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:40975
--local_ip_for_outbound_sockets=127.24.216.130
--tserver_master_addrs=127.24.216.190:36699
--webserver_port=39515
--webserver_interface=127.24.216.130
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.216.148:49043/kms with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:15:16 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648116, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:16.908442 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:16.880219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36699 (local address 127.24.216.129:44705)
0929 02:15:16.881691 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:16.881745 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:16.884217 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:16.887283 (+ 3066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:16.887294 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:16.887955 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:16.888757 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:16.888788 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:16.891138 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:16.891143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:16.892474 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:16.892486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:16.894218 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:16.894227 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:16.894515 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:16.895462 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:16.895495 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:16.898268 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:16.902968 (+ 4700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:16.902979 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:16.902998 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:16.903638 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:16.904200 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:16.904205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:16.904209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:16.904491 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:16.905629 (+ 1138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:16.905637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:16.906072 (+ 435us) client_negotiation.cc:769] Sending connection context
0929 02:15:16.906452 (+ 380us) client_negotiation.cc:240] Negotiation successful
0929 02:15:16.907042 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"thread_start_us":334,"threads_started":1}
I20260929 02:15:16.908468 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:16.880252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:44705 (local address 127.24.216.190:36699)
0929 02:15:16.881364 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:16.881383 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:16.881900 (+ 517us) server_negotiation.cc:407] Connection header received
0929 02:15:16.885089 (+ 3189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:16.885108 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:16.885724 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:16.886503 (+ 779us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:16.889037 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:16.890939 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:16.892631 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:16.893618 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:16.893802 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:16.898628 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:16.898640 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:16.898660 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:16.898713 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:16.902781 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:16.903810 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:16.903821 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:16.903829 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:16.904043 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:16.904643 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:16.904648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:16.904652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:16.905334 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:16.905540 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:16.906720 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0929 02:15:16.907146 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":615,"threads_started":1}
I20260929 02:15:16.912251 6028 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36699
I20260929 02:15:16.912633 6028 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:16.913590 6028 heartbeater.cc:511] Master 127.24.216.190:36699 requested a full tablet report, sending...
W20260929 02:15:16.916908 6028 heartbeater.cc:644] Failed to heartbeat to 127.24.216.190:36699 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 02:15:16.917152 6028 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:16.917457 6028 heartbeater.cc:511] Master 127.24.216.190:36699 requested a full tablet report, sending...
I20260929 02:15:16.918790 6028 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:16.919140 6028 heartbeater.cc:511] Master 127.24.216.190:36699 requested a full tablet report, sending...
W20260929 02:15:16.920303 6028 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:15:17.360190 6032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:17.360529 6032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:17.360718 6032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:17.372394 6032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:17.372632 6032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:17.372862 6032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:15:17.389586 6032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.216.148:49043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:40975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=39515
--tserver_master_addrs=127.24.216.190:36699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:17.391644 6032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:17.393499 6032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:17.411691 6048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:17.412102 6049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:17.420800 6051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:17.421859 6032 server_base.cc:1034] running on GCE node
I20260929 02:15:17.422813 6032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:17.424220 6032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:17.425539 6032 hybrid_clock.cc:648] HybridClock initialized: now 1790648117425466 us; error 64 us; skew 500 ppm
I20260929 02:15:17.425982 6032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:17 dist-test-slave-9k0h krb5kdc[4819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648117, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:17.430861 6032 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:15:17.433195 6032 webserver.cc:460] Webserver started at http://127.24.216.130:39515/ using document root <none> and password file <none>
I20260929 02:15:17.434073 6032 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:17.434315 6032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:15:17 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648117, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
Sep 29 02:15:17 dist-test-slave-9k0h krb5kdc[4819](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790648117, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:17.458385 6032 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:15:17.474433 6059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:17.480089 6032 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.003s sys 0.012s
I20260929 02:15:17.480430 6032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8"
format_stamp: "Formatted at 2026-09-29 02:15:14 on dist-test-slave-9k0h"
server_key: "82b0c2ba2e41ac9f63595853eab65c2c"
server_key_iv: "0f01829c2c981a7a30a044f7f709170d"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:17.482070 6032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:17.508843 6032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:17.513752 6032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:17.514050 6032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:17.515216 6032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:17.516683 6032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:17.516772 6032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:17.516906 6032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:17.517007 6032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:17.583714 6032 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:40975
I20260929 02:15:17.583803 6172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:40975 every 8 connection(s)
I20260929 02:15:17.585619 6032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:15:17.593915 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 6032
I20260929 02:15:17.594910 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.216.148:49043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:36315
--local_ip_for_outbound_sockets=127.24.216.131
--tserver_master_addrs=127.24.216.190:36699
--webserver_port=32973
--webserver_interface=127.24.216.131
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:35817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.216.148:49043/kms with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:15:17 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648117, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:17.659286 6176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:17.590539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:49577 (local address 127.24.216.190:36699)
0929 02:15:17.621867 (+ 31328us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:17.621874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:17.621904 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:15:17.621984 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:17.621990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:17.622106 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:17.622281 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:17.624842 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:17.626093 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:17.628974 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:17.629796 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:17.629943 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:17.638903 (+ 8960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:17.638913 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:17.638919 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:17.638980 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:17.645704 (+ 6724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:17.646979 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:17.646985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:17.646989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:17.647158 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:17.650637 (+ 3479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:17.650644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:17.650648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:17.651043 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:17.651182 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:17.658486 (+ 7304us) server_negotiation.cc:299] Negotiation successful
0929 02:15:17.658816 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28706,"thread_start_us":28591,"threads_started":1}
I20260929 02:15:17.661813 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:17.589751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36699 (local address 127.24.216.130:49577)
0929 02:15:17.614867 (+ 25116us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:17.614936 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:17.618436 (+ 3500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:17.622834 (+ 4398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:17.622851 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:17.623625 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:17.624586 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:17.624618 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:17.626360 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:17.626368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:17.628681 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:17.628711 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:17.630341 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:17.630350 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:17.630721 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:17.632288 (+ 1567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:17.632335 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:17.638486 (+ 6151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:17.645967 (+ 7481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:17.645979 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:17.646003 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:17.646782 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:17.650079 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:17.650085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:17.650089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:17.650473 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:17.651307 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:17.651318 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:17.657759 (+ 6441us) client_negotiation.cc:769] Sending connection context
0929 02:15:17.659697 (+ 1938us) client_negotiation.cc:240] Negotiation successful
0929 02:15:17.660339 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24762,"thread_start_us":24433,"threads_started":1}
I20260929 02:15:17.663674 6173 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36699
I20260929 02:15:17.664006 6173 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:17.664945 6173 heartbeater.cc:511] Master 127.24.216.190:36699 requested a full tablet report, sending...
I20260929 02:15:17.667171 5831 ts_manager.cc:194] Registered new tserver with Master: df1d6d5f9096496c85b2d2a4e0bb1ed8 (127.24.216.130:40975)
I20260929 02:15:17.669484 5831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:49577
I20260929 02:15:17.920509 6028 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:17.920970 6028 heartbeater.cc:511] Master 127.24.216.190:36699 requested a full tablet report, sending...
I20260929 02:15:17.922224 5831 ts_manager.cc:194] Registered new tserver with Master: 9bf396a65a2b41eaadc27427f0de83bc (127.24.216.129:34191)
I20260929 02:15:17.923406 5831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:44705
Sep 29 02:15:17 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648117, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
W20260929 02:15:18.095569 6177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:18.095912 6177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:18.096087 6177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:18.104168 6177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:18.104334 6177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:18.104526 6177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:15:18.121151 6177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.216.148:49043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:36315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=32973
--tserver_master_addrs=127.24.216.190:36699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:18.122938 6177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:18.124635 6177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:18.170210 6182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:18.178511 6183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:18.193547 6186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:18.194540 6177 server_base.cc:1034] running on GCE node
I20260929 02:15:18.195863 6177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:18.204691 6177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:18.206290 6177 hybrid_clock.cc:648] HybridClock initialized: now 1790648118206178 us; error 117 us; skew 500 ppm
I20260929 02:15:18.206822 6177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:18 dist-test-slave-9k0h krb5kdc[4819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648118, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:18.211862 6177 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:15:18.250675 6177 webserver.cc:460] Webserver started at http://127.24.216.131:32973/ using document root <none> and password file <none>
I20260929 02:15:18.252056 6177 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:18.252336 6177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:15:18 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648118, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
I20260929 02:15:18.298372 6177 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.006s sys 0.004s
I20260929 02:15:18.308914 6193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:18.311035 6177 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.004s
I20260929 02:15:18.311372 6177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "ba91392c21a3453fafc548fda59e9d4e"
format_stamp: "Formatted at 2026-09-29 02:15:14 on dist-test-slave-9k0h"
server_key: "27c390719ba1e1b3b370b072651025ce"
server_key_iv: "28af955a4e990556b868bcfb01ac5651"
server_key_version: "kuduclusterkey@0"
I20260929 02:15:18.312855 6177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:18.358002 6177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 29 02:15:18 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648117, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for HTTP/127.24.216.148@KRBTEST.COM
I20260929 02:15:18.365917 6177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:18.366277 6177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:18.390637 6177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:18.398918 6177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:18.399063 6177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:18.399183 6177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:18.399258 6177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:18.675307 6173 heartbeater.cc:503] Master 127.24.216.190:36699 was elected leader, sending a full tablet report...
I20260929 02:15:18.816880 6177 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:36315
I20260929 02:15:18.816902 6313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:36315 every 8 connection(s)
I20260929 02:15:18.818594 6177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:15:18.822965 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 6177
Sep 29 02:15:18 dist-test-slave-9k0h krb5kdc[4819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648118, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:18.861162 6317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:18.824784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.24.216.190:36699)
0929 02:15:18.841820 (+ 17036us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:18.841826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:18.841847 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:15:18.841912 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:18.841918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:18.842014 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:18.842123 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:18.844313 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.845538 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.847450 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.848115 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.848250 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:18.852192 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:18.852199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:18.852203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:18.852264 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:18.855906 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:18.856895 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:18.856900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:18.856904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:18.857066 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:18.857815 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:18.857820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:18.857823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:18.858217 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:18.858479 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:18.860516 (+ 2037us) server_negotiation.cc:299] Negotiation successful
0929 02:15:18.860759 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16914,"thread_start_us":16791,"threads_started":1}
I20260929 02:15:18.869462 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:18.841800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:36211 (local address 127.24.216.190:36699)
0929 02:15:18.848413 (+ 6613us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:18.848420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:18.848452 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:15:18.848509 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:18.848514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:18.848601 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:18.851211 (+ 2610us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:18.851473 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.852738 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.855662 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.856274 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.857298 (+ 1024us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:18.860873 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:18.860928 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:18.860934 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:18.860976 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:18.865525 (+ 4549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:18.866585 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:18.866591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:18.866594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:18.866735 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:18.867291 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:18.867297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:18.867300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:18.867633 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:18.867783 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:18.868779 (+ 996us) server_negotiation.cc:299] Negotiation successful
0929 02:15:18.869017 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6444,"thread_start_us":6933,"threads_started":1}
I20260929 02:15:18.870760 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:18.826106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36699 (local address 127.24.216.131:36211)
0929 02:15:18.835003 (+ 8897us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:18.835080 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:18.840328 (+ 5248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:18.849298 (+ 8970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:18.849314 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:18.850017 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:18.850996 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:18.851026 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.852963 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.852968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:18.855038 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:18.855071 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.856554 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.856561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:18.856869 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:18.857584 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:18.857615 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:18.860524 (+ 2909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:18.865703 (+ 5179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:18.865714 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:18.865735 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:18.866430 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:18.866862 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:18.866867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:18.866872 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:18.867155 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:18.867791 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:18.867799 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:18.868299 (+ 500us) client_negotiation.cc:769] Sending connection context
0929 02:15:18.868644 (+ 345us) client_negotiation.cc:240] Negotiation successful
0929 02:15:18.869244 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8394,"thread_start_us":11884,"threads_started":1}
I20260929 02:15:18.872762 6314 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36699
I20260929 02:15:18.873097 6314 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:18.873989 6314 heartbeater.cc:511] Master 127.24.216.190:36699 requested a full tablet report, sending...
I20260929 02:15:18.876700 5831 ts_manager.cc:194] Registered new tserver with Master: ba91392c21a3453fafc548fda59e9d4e (127.24.216.131:36315)
I20260929 02:15:18.877816 5831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:36211
I20260929 02:15:18.885015 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:15:18.908113 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:18.889962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.24.216.190:36699)
0929 02:15:18.890185 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:18.890192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:18.890888 (+ 696us) server_negotiation.cc:407] Connection header received
0929 02:15:18.891106 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:18.891113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:18.891213 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:18.891323 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:18.893300 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.894419 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.896317 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.896876 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.897073 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:18.900565 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:18.900572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:18.900577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:18.900624 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:18.904506 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:18.905629 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:18.905636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:18.905640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:18.905818 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:18.906401 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:18.906407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:18.906410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:18.906805 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:18.907058 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:18.907518 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 02:15:18.907777 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:15:18.915512 5831 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 02:15:18.926833 6028 heartbeater.cc:503] Master 127.24.216.190:36699 was elected leader, sending a full tablet report...
W20260929 02:15:18.928092 5831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:15:18.983667 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:18.948519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:34191 (local address 127.0.0.1:56960)
0929 02:15:18.971153 (+ 22634us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:18.971176 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:18.971326 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:18.974211 (+ 2885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:18.974216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:18.974246 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:18.974620 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:18.974631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.976892 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.976898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:18.979883 (+ 2985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:18.979895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.982515 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.982523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:18.982777 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:18.982845 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 02:15:18.982979 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 02:15:18.983203 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22488,"mutex_wait_us":312,"thread_start_us":24242,"threads_started":1}
I20260929 02:15:18.986413 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:18.948481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:40975 (local address 127.0.0.1:37338)
0929 02:15:18.962966 (+ 14485us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:18.963038 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:18.963288 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:18.972471 (+ 9183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:18.972481 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:18.972525 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:18.973014 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:18.973039 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.979472 (+ 6433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.979481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:18.981526 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:18.981553 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.985576 (+ 4023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.985583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:18.985773 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:18.985810 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:15:18.985882 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 02:15:18.986030 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14090,"thread_start_us":22377,"threads_started":1}
I20260929 02:15:18.987973 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:18.949191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.24.216.130:40975)
0929 02:15:18.971668 (+ 22477us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:18.971678 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:18.971713 (+ 35us) server_negotiation.cc:407] Connection header received
0929 02:15:18.971826 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:18.971835 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:18.972190 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:18.972460 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:18.973303 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.975097 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.983319 (+ 8222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.985328 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.987050 (+ 1722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:18.987252 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:18.987332 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 02:15:18.987573 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22263,"thread_start_us":22221,"threads_started":1}
I20260929 02:15:18.988054 6334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:18.957780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.24.216.131:36315)
0929 02:15:18.964104 (+ 6324us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:18.964119 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:18.972271 (+ 8152us) server_negotiation.cc:407] Connection header received
0929 02:15:18.972648 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:18.972661 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:18.973002 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:18.973188 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:18.975326 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.976993 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.984197 (+ 7204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.985991 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.986207 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:18.986403 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:18.987390 (+ 987us) server_negotiation.cc:299] Negotiation successful
0929 02:15:18.987574 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6078,"thread_start_us":5895,"threads_started":1}
I20260929 02:15:18.988978 6329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:18.956722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:36315 (local address 127.0.0.1:46206)
0929 02:15:18.972130 (+ 15408us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:18.972151 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:18.972400 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:18.974753 (+ 2353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:18.974757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:18.974782 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:18.975120 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:18.975131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.977182 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.977187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:18.979704 (+ 2517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:18.979726 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.986960 (+ 7234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.986967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:18.987159 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:18.987195 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 02:15:18.988372 (+ 1177us) client_negotiation.cc:240] Negotiation successful
0929 02:15:18.988537 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15265,"mutex_wait_us":241,"thread_start_us":5836,"threads_started":1}
I20260929 02:15:18.992933 6242 tablet_service.cc:1467] Processing CreateTablet for tablet 22905588926e46e8a75f06240108905b (DEFAULT_TABLE table=test-table [id=d1d5cfe1aee84a21b338620fdb1ff336]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:15:18.992942 6108 tablet_service.cc:1467] Processing CreateTablet for tablet 22905588926e46e8a75f06240108905b (DEFAULT_TABLE table=test-table [id=d1d5cfe1aee84a21b338620fdb1ff336]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:15:18.994565 6242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22905588926e46e8a75f06240108905b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:18.994585 6108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22905588926e46e8a75f06240108905b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:19.003341 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:18.948752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56960 (local address 127.24.216.129:34191)
0929 02:15:18.973470 (+ 24718us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:18.973481 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:18.973509 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:15:18.973623 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:18.973631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:18.973884 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:18.974800 (+ 916us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:18.975039 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.976712 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:18.980072 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:18.981760 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.002432 (+ 20672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:19.002686 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:19.002781 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 02:15:19.002981 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24515,"thread_start_us":24359,"threads_started":1}
I20260929 02:15:19.008011 5963 tablet_service.cc:1467] Processing CreateTablet for tablet 22905588926e46e8a75f06240108905b (DEFAULT_TABLE table=test-table [id=d1d5cfe1aee84a21b338620fdb1ff336]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:15:19.010244 5963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22905588926e46e8a75f06240108905b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:19.023015 6335 tablet_bootstrap.cc:492] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8: Bootstrap starting.
I20260929 02:15:19.027267 6335 tablet_bootstrap.cc:654] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:19.028623 6335 log.cc:824] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:19.030956 6336 tablet_bootstrap.cc:492] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e: Bootstrap starting.
I20260929 02:15:19.034719 6336 tablet_bootstrap.cc:654] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:19.039403 6338 tablet_bootstrap.cc:492] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc: Bootstrap starting.
I20260929 02:15:19.039880 6336 log.cc:824] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:19.043249 6338 tablet_bootstrap.cc:654] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:19.043866 6335 tablet_bootstrap.cc:492] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8: No bootstrap required, opened a new log
I20260929 02:15:19.044174 6335 ts_tablet_manager.cc:1397] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.000s
I20260929 02:15:19.044776 6338 log.cc:824] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:19.046067 6336 tablet_bootstrap.cc:492] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e: No bootstrap required, opened a new log
I20260929 02:15:19.046506 6336 ts_tablet_manager.cc:1397] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260929 02:15:19.054358 6335 raft_consensus.cc:348] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40975 } } peers { permanent_uuid: "9bf396a65a2b41eaadc27427f0de83bc" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 34191 } } peers { permanent_uuid: "ba91392c21a3453fafc548fda59e9d4e" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36315 } }
I20260929 02:15:19.054768 6338 tablet_bootstrap.cc:492] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc: No bootstrap required, opened a new log
I20260929 02:15:19.054842 6335 raft_consensus.cc:374] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:19.054931 6335 raft_consensus.cc:729] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1d6d5f9096496c85b2d2a4e0bb1ed8, State: Initialized, Role: FOLLOWER
I20260929 02:15:19.055074 6338 ts_tablet_manager.cc:1397] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260929 02:15:19.055661 6335 consensus_queue.cc:260] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40975 } } peers { permanent_uuid: "9bf396a65a2b41eaadc27427f0de83bc" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 34191 } } peers { permanent_uuid: "ba91392c21a3453fafc548fda59e9d4e" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36315 } }
I20260929 02:15:19.063943 6335 ts_tablet_manager.cc:1428] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
I20260929 02:15:19.065404 6338 raft_consensus.cc:348] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40975 } } peers { permanent_uuid: "9bf396a65a2b41eaadc27427f0de83bc" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 34191 } } peers { permanent_uuid: "ba91392c21a3453fafc548fda59e9d4e" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36315 } }
I20260929 02:15:19.065908 6338 raft_consensus.cc:374] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:19.066013 6338 raft_consensus.cc:729] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf396a65a2b41eaadc27427f0de83bc, State: Initialized, Role: FOLLOWER
I20260929 02:15:19.066810 6338 consensus_queue.cc:260] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40975 } } peers { permanent_uuid: "9bf396a65a2b41eaadc27427f0de83bc" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 34191 } } peers { permanent_uuid: "ba91392c21a3453fafc548fda59e9d4e" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36315 } }
I20260929 02:15:19.071506 6336 raft_consensus.cc:348] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40975 } } peers { permanent_uuid: "9bf396a65a2b41eaadc27427f0de83bc" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 34191 } } peers { permanent_uuid: "ba91392c21a3453fafc548fda59e9d4e" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36315 } }
I20260929 02:15:19.072300 6336 raft_consensus.cc:374] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:19.072472 6336 raft_consensus.cc:729] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba91392c21a3453fafc548fda59e9d4e, State: Initialized, Role: FOLLOWER
I20260929 02:15:19.073470 6336 consensus_queue.cc:260] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40975 } } peers { permanent_uuid: "9bf396a65a2b41eaadc27427f0de83bc" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 34191 } } peers { permanent_uuid: "ba91392c21a3453fafc548fda59e9d4e" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36315 } }
I20260929 02:15:19.075368 6338 ts_tablet_manager.cc:1428] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc: Time spent starting tablet: real 0.020s user 0.019s sys 0.000s
I20260929 02:15:19.082420 6314 heartbeater.cc:503] Master 127.24.216.190:36699 was elected leader, sending a full tablet report...
I20260929 02:15:19.083233 6336 ts_tablet_manager.cc:1428] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e: Time spent starting tablet: real 0.037s user 0.017s sys 0.004s
W20260929 02:15:19.090339 6315 tablet.cc:2369] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:15:19.130420 6029 tablet.cc:2369] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:15:19.132104 6174 tablet.cc:2369] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:15:19.311013 6341 raft_consensus.cc:482] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:15:19.311395 6341 raft_consensus.cc:504] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40975 } } peers { permanent_uuid: "9bf396a65a2b41eaadc27427f0de83bc" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 34191 } } peers { permanent_uuid: "ba91392c21a3453fafc548fda59e9d4e" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36315 } }
I20260929 02:15:19.313272 6341 leader_election.cc:288] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bf396a65a2b41eaadc27427f0de83bc (127.24.216.129:34191), ba91392c21a3453fafc548fda59e9d4e (127.24.216.131:36315)
I20260929 02:15:19.320788 6334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.313769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:39385 (local address 127.24.216.131:36315)
0929 02:15:19.314005 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:19.314013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:19.314700 (+ 687us) server_negotiation.cc:407] Connection header received
0929 02:15:19.314925 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:19.314930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:19.314993 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:19.315078 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:19.315786 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.316555 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.318650 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.319728 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.319855 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:19.319911 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:19.320368 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 02:15:19.320503 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:15:19.321002 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.313813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:53573 (local address 127.24.216.129:34191)
0929 02:15:19.314032 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:19.314042 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:19.314692 (+ 650us) server_negotiation.cc:407] Connection header received
0929 02:15:19.314946 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:19.314950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:19.315017 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:19.315139 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:19.315910 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.316595 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.318797 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.320059 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.320215 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:19.320286 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:19.320611 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 02:15:19.320750 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:15:19.321300 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.313636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:36315 (local address 127.24.216.130:39385)
0929 02:15:19.314562 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:19.314592 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:19.314751 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:19.315133 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:19.315138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:19.315171 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:19.315608 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:19.315628 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.316688 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.316694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:19.318491 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:19.318507 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.319990 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.319998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:19.320186 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:19.320228 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 02:15:19.320326 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 02:15:19.320435 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":24,"thread_start_us":412,"threads_started":1}
I20260929 02:15:19.321336 6345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.313716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:34191 (local address 127.24.216.130:53573)
0929 02:15:19.314562 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:19.314583 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:19.314771 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:19.315201 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:19.315206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:19.315240 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:19.315736 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:19.315756 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.316758 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.316761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:19.318634 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:19.318647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.320252 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.320257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:19.320429 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:19.320462 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:15:19.320558 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 02:15:19.320692 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":76,"thread_start_us":494,"threads_started":1}
I20260929 02:15:19.322512 5983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22905588926e46e8a75f06240108905b" candidate_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf396a65a2b41eaadc27427f0de83bc" is_pre_election: true
I20260929 02:15:19.322510 6269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22905588926e46e8a75f06240108905b" candidate_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba91392c21a3453fafc548fda59e9d4e" is_pre_election: true
I20260929 02:15:19.323099 6269 raft_consensus.cc:2454] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1d6d5f9096496c85b2d2a4e0bb1ed8 in term 0.
I20260929 02:15:19.323099 5983 raft_consensus.cc:2454] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1d6d5f9096496c85b2d2a4e0bb1ed8 in term 0.
I20260929 02:15:19.323853 6062 leader_election.cc:302] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9bf396a65a2b41eaadc27427f0de83bc, ba91392c21a3453fafc548fda59e9d4e, df1d6d5f9096496c85b2d2a4e0bb1ed8; no voters:
I20260929 02:15:19.324404 6341 raft_consensus.cc:2781] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:15:19.324568 6341 raft_consensus.cc:482] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:15:19.324717 6341 raft_consensus.cc:3037] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:19.327039 6341 raft_consensus.cc:504] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40975 } } peers { permanent_uuid: "9bf396a65a2b41eaadc27427f0de83bc" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 34191 } } peers { permanent_uuid: "ba91392c21a3453fafc548fda59e9d4e" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36315 } }
I20260929 02:15:19.327642 6341 leader_election.cc:288] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [CANDIDATE]: Term 1 election: Requested vote from peers 9bf396a65a2b41eaadc27427f0de83bc (127.24.216.129:34191), ba91392c21a3453fafc548fda59e9d4e (127.24.216.131:36315)
I20260929 02:15:19.328224 6269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22905588926e46e8a75f06240108905b" candidate_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba91392c21a3453fafc548fda59e9d4e"
I20260929 02:15:19.328222 5983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22905588926e46e8a75f06240108905b" candidate_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf396a65a2b41eaadc27427f0de83bc"
I20260929 02:15:19.328444 6269 raft_consensus.cc:3037] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:19.328461 5983 raft_consensus.cc:3037] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:19.330704 6269 raft_consensus.cc:2454] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1d6d5f9096496c85b2d2a4e0bb1ed8 in term 1.
I20260929 02:15:19.330745 5983 raft_consensus.cc:2454] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1d6d5f9096496c85b2d2a4e0bb1ed8 in term 1.
I20260929 02:15:19.331197 6063 leader_election.cc:302] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba91392c21a3453fafc548fda59e9d4e, df1d6d5f9096496c85b2d2a4e0bb1ed8; no voters:
I20260929 02:15:19.331502 6341 raft_consensus.cc:2781] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:15:19.331977 6341 raft_consensus.cc:686] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [term 1 LEADER]: Becoming Leader. State: Replica: df1d6d5f9096496c85b2d2a4e0bb1ed8, State: Running, Role: LEADER
I20260929 02:15:19.332437 6341 consensus_queue.cc:237] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40975 } } peers { permanent_uuid: "9bf396a65a2b41eaadc27427f0de83bc" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 34191 } } peers { permanent_uuid: "ba91392c21a3453fafc548fda59e9d4e" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36315 } }
I20260929 02:15:19.337327 5831 catalog_manager.cc:5495] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 reported cstate change: term changed from 0 to 1, leader changed from <none> to df1d6d5f9096496c85b2d2a4e0bb1ed8 (127.24.216.130). New cstate: current_term: 1 leader_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1d6d5f9096496c85b2d2a4e0bb1ed8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bf396a65a2b41eaadc27427f0de83bc" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 34191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba91392c21a3453fafc548fda59e9d4e" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36315 } health_report { overall_health: UNKNOWN } } }
I20260929 02:15:19.421705 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.415449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.24.216.130:40975)
0929 02:15:19.415633 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:19.415638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:19.415994 (+ 356us) server_negotiation.cc:407] Connection header received
0929 02:15:19.416200 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:19.416209 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:19.416310 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:19.416449 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:15:19.416821 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.417586 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.419245 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.419766 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.419872 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:19.420423 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:15:19.421027 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:15:19.421162 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:19.421334 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 02:15:19.421407 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:15:19.430751 5983 raft_consensus.cc:1261] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc [term 1 FOLLOWER]: Refusing update from remote peer df1d6d5f9096496c85b2d2a4e0bb1ed8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:15:19.430749 6269 raft_consensus.cc:1261] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e [term 1 FOLLOWER]: Refusing update from remote peer df1d6d5f9096496c85b2d2a4e0bb1ed8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:15:19.431936 6346 consensus_queue.cc:1035] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf396a65a2b41eaadc27427f0de83bc" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 34191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:15:19.432305 6341 consensus_queue.cc:1035] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba91392c21a3453fafc548fda59e9d4e" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:15:19.459591 5830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:15:19.459937 5830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:15:19.464262 5830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc3d8681bfeb4abab7ccf5f8da465dfb: Sending DeleteTablet for 3 replicas of tablet 22905588926e46e8a75f06240108905b
I20260929 02:15:19.465564 6108 tablet_service.cc:1514] Processing DeleteTablet for tablet 22905588926e46e8a75f06240108905b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:15:19 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:37338
I20260929 02:15:19.465559 6242 tablet_service.cc:1514] Processing DeleteTablet for tablet 22905588926e46e8a75f06240108905b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:15:19 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:46206
I20260929 02:15:19.465564 5963 tablet_service.cc:1514] Processing DeleteTablet for tablet 22905588926e46e8a75f06240108905b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:15:19 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:56960
I20260929 02:15:19.471984 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 5868
I20260929 02:15:19.478435 6362 tablet_replica.cc:330] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8: stopping tablet replica
I20260929 02:15:19.478718 6361 tablet_replica.cc:330] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e: stopping tablet replica
I20260929 02:15:19.479091 6361 raft_consensus.cc:2229] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:15:19.479094 6362 raft_consensus.cc:2229] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:15:19.479782 6361 raft_consensus.cc:2258] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:15:19.480779 6360 tablet_replica.cc:330] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc: stopping tablet replica
I20260929 02:15:19.482668 6360 raft_consensus.cc:2229] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:15:19.482949 6361 ts_tablet_manager.cc:1905] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:15:19.483294 6360 raft_consensus.cc:2258] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:15:19.485426 6360 ts_tablet_manager.cc:1905] T 22905588926e46e8a75f06240108905b P 9bf396a65a2b41eaadc27427f0de83bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:15:19.486167 6362 raft_consensus.cc:2258] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:15:19.489621 6362 ts_tablet_manager.cc:1905] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:15:19.492532 6361 ts_tablet_manager.cc:1918] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:15:19.492880 6361 log.cc:1196] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/22905588926e46e8a75f06240108905b
I20260929 02:15:19.493865 6361 ts_tablet_manager.cc:1939] T 22905588926e46e8a75f06240108905b P ba91392c21a3453fafc548fda59e9d4e: Deleting consensus metadata
I20260929 02:15:19.496932 5818 catalog_manager.cc:4841] TS ba91392c21a3453fafc548fda59e9d4e (127.24.216.131:36315): tablet 22905588926e46e8a75f06240108905b (table test-table [id=d1d5cfe1aee84a21b338620fdb1ff336]) successfully deleted
I20260929 02:15:19.499277 6362 ts_tablet_manager.cc:1918] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:15:19.499493 6362 log.cc:1196] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/22905588926e46e8a75f06240108905b
I20260929 02:15:19.500002 6362 ts_tablet_manager.cc:1939] T 22905588926e46e8a75f06240108905b P df1d6d5f9096496c85b2d2a4e0bb1ed8: Deleting consensus metadata
I20260929 02:15:19.501868 5815 catalog_manager.cc:4841] TS df1d6d5f9096496c85b2d2a4e0bb1ed8 (127.24.216.130:40975): tablet 22905588926e46e8a75f06240108905b (table test-table [id=d1d5cfe1aee84a21b338620fdb1ff336]) successfully deleted
I20260929 02:15:19.502429 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.473540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56964 (local address 127.24.216.129:34191)
0929 02:15:19.473729 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:19.473735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:19.480684 (+ 6949us) server_negotiation.cc:407] Connection header received
0929 02:15:19.480914 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:19.480925 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:19.481013 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:19.481150 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:19.482655 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.483842 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.486263 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.487011 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.487136 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:19.494511 (+ 7375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:19.494533 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:19.494556 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:19.494614 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:19.498526 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:19.499572 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:19.499583 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:19.499587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:19.499851 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:19.500469 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:19.500475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:19.500480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:19.500933 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:19.501246 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:19.501640 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 02:15:19.501906 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:15:19.503410 6023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:19.595691 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 5868
W20260929 02:15:19.614728 5817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.216.129:34191 (error 108)
I20260929 02:15:19.614840 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 6032
I20260929 02:15:19.618985 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.616988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:34191 (local address 127.0.0.1:56968)
0929 02:15:19.618598 (+ 1610us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:19.618732 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1480,"thread_start_us":1366,"threads_started":1}
W20260929 02:15:19.619899 5817 catalog_manager.cc:4584] TS 9bf396a65a2b41eaadc27427f0de83bc (127.24.216.129:34191): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 22905588926e46e8a75f06240108905b: Network error: Client connection negotiation failed: client connection to 127.24.216.129:34191: connect: Connection refused (error 111)
I20260929 02:15:19.630627 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.615743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.24.216.130:40975)
0929 02:15:19.615992 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:19.616002 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:19.616027 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:15:19.616215 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:19.616223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:19.616323 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:19.616437 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:19.618679 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.619627 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.621143 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.621611 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.621751 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:19.624368 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:19.624381 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:19.624396 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:19.624433 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:19.627660 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:19.628461 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:19.628468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:19.628470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:19.628608 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:19.628998 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:19.629004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:19.629009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:19.629364 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:19.629672 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:19.629981 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 02:15:19.630249 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:15:19.631302 6168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:19.673992 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.673400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:34191 (local address 127.0.0.1:56980)
0929 02:15:19.673648 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:19.673741 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 02:15:19.675096 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.674740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:34191 (local address 127.0.0.1:56984)
0929 02:15:19.674868 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:19.674964 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 02:15:19.735775 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.735258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:34191 (local address 127.0.0.1:56994)
0929 02:15:19.735502 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:19.735612 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 02:15:19.737016 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.736606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:34191 (local address 127.0.0.1:56998)
0929 02:15:19.736779 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:19.736853 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 02:15:19.746876 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 6032
I20260929 02:15:19.766386 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 6177
I20260929 02:15:19.780572 6334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.767199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46222 (local address 127.24.216.131:36315)
0929 02:15:19.767405 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:19.767409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:19.767427 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:15:19.767579 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:19.767582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:19.767643 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:19.767734 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:19.768974 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.769747 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.771153 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:19.771572 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:19.771671 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:19.774486 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:19.774500 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:19.774516 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:19.774550 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:19.777392 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:19.778296 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:19.778304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:19.778306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:19.778445 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:19.778781 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:19.778784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:19.778786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:19.779161 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:19.779543 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:19.779899 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 02:15:19.780135 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 02:15:19.781289 6309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:19.817869 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.817343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:34191 (local address 127.0.0.1:57010)
0929 02:15:19.817606 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:19.817702 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 02:15:19.819380 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:19.818746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:34191 (local address 127.0.0.1:57012)
0929 02:15:19.818993 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:19.819215 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 02:15:19.872094 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 6177
I20260929 02:15:19.889845 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 5798
I20260929 02:15:19.891142 5861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:20.007807 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 5798
I20260929 02:15:20.007929 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:20.007317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:34191 (local address 127.0.0.1:57024)
0929 02:15:20.007543 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:20.007703 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 02:15:20.036389 25442 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 02:15:20.382706 25442 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 02:15:20.382911 25442 mini_ranger.cc:66] Stopping Ranger...
I20260929 02:15:20.729326 25442 mini_ranger.cc:68] Stopped Ranger
2026-09-29 02:15:20.729 UTC [4849] LOG: received smart shutdown request
2026-09-29 02:15:20.733 UTC [4849] LOG: background worker "logical replication launcher" (PID 4858) exited with exit code 1
2026-09-29 02:15:20.733 UTC [4853] LOG: shutting down
2026-09-29 02:15:20.930 UTC [4849] LOG: database system is shut down
2026-09-29T02:15:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55337 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:15:21 dist-test-slave-9k0h krb5kdc[6378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:15:21 dist-test-slave-9k0h krb5kdc[6378](info): set up 2 sockets
Sep 29 02:15:21 dist-test-slave-9k0h krb5kdc[6378](info): commencing operation
krb5kdc: starting...
W20260929 02:15:22.206875 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:15:22 dist-test-slave-9k0h krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:15:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:15:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:15:22.474794 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:45717
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:40745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:45717,127.24.216.189:41583,127.24.216.188:45869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:22.813161 6391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:22.813473 6391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:22.813627 6391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:22.821173 6391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:15:22.821301 6391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:22.821408 6391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:22.821519 6391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:15:22.821610 6391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:15:22.832403 6391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:40745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:45717,127.24.216.189:41583,127.24.216.188:45869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:45717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:22.833951 6391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:22.835500 6391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:22.848325 6396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:22.848294 6399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:22.848507 6397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:22.849359 6391 server_base.cc:1034] running on GCE node
I20260929 02:15:22.850323 6391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:22.851863 6391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:22.853111 6391 hybrid_clock.cc:648] HybridClock initialized: now 1790648122853063 us; error 36 us; skew 500 ppm
I20260929 02:15:22.853552 6391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:22 dist-test-slave-9k0h krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648122, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:22.856905 6391 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:15:22.858611 6391 webserver.cc:460] Webserver started at http://127.24.216.190:33247/ using document root <none> and password file <none>
I20260929 02:15:22.859351 6391 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:22.859509 6391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:22.859853 6391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:15:22.862993 6391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "a73e32944be943b1849653bc54ba1b33"
format_stamp: "Formatted at 2026-09-29 02:15:22 on dist-test-slave-9k0h"
server_key: "f038610537faf95d9b8937c1fa5c7d37"
server_key_iv: "24e5c21c5abd31a6e7a393f155a0e152"
server_key_version: "encryptionkey@0"
I20260929 02:15:22.863817 6391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "a73e32944be943b1849653bc54ba1b33"
format_stamp: "Formatted at 2026-09-29 02:15:22 on dist-test-slave-9k0h"
server_key: "f038610537faf95d9b8937c1fa5c7d37"
server_key_iv: "24e5c21c5abd31a6e7a393f155a0e152"
server_key_version: "encryptionkey@0"
I20260929 02:15:22.869164 6391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:15:22.872704 6406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:22.874038 6391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:15:22.874279 6391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "a73e32944be943b1849653bc54ba1b33"
format_stamp: "Formatted at 2026-09-29 02:15:22 on dist-test-slave-9k0h"
server_key: "f038610537faf95d9b8937c1fa5c7d37"
server_key_iv: "24e5c21c5abd31a6e7a393f155a0e152"
server_key_version: "encryptionkey@0"
I20260929 02:15:22.874488 6391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:22.889842 6391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:22.894345 6391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:22.894654 6391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:22.913254 6391 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:45717
I20260929 02:15:22.913276 6457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:45717 every 8 connection(s)
I20260929 02:15:22.914688 6391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:15:22.918397 6458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:22.920204 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 6391
I20260929 02:15:22.920428 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:15:22.921268 25442 external_mini_cluster.cc:1428] Setting key da124b2f1dd0d377b1a31debd076571d
I20260929 02:15:22.928236 6458 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:15:22 dist-test-slave-9k0h krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648122, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.216.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.189@KRBTEST.COM" created.
I20260929 02:15:22.952791 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:22.932461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.24.216.190:45717)
0929 02:15:22.933359 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:22.933379 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:22.933800 (+ 421us) server_negotiation.cc:407] Connection header received
0929 02:15:22.935908 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:22.935914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:22.936162 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:22.936376 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:22.937672 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:22.938926 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:22.940331 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:22.940888 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:22.941109 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:22.944859 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:22.944867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:22.944880 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:22.944911 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:22.947970 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:22.948779 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:22.948786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:22.948788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:22.948950 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:22.949461 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:22.949469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:22.949472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:22.950029 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:22.950289 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:22.950832 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 02:15:22.951255 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":639,"thread_start_us":392,"threads_started":1}
I20260929 02:15:22.952812 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:22.932481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:45717 (local address 127.0.0.1:47502)
0929 02:15:22.933641 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:22.933674 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:22.935526 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:22.936390 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:22.936397 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:22.937127 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:22.937466 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:22.937484 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:22.939085 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:22.939093 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:22.940188 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:22.940199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:22.941124 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:22.941130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:22.941296 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:22.941996 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:22.942020 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:22.944613 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:22.948135 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:22.948146 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:22.948156 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:22.948594 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:22.949069 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:22.949076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:22.949081 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:22.949321 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:22.950208 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:22.950216 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:22.950436 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 02:15:22.950699 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 02:15:22.951363 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":203,"thread_start_us":754,"threads_started":1}
I20260929 02:15:22.956056 6458 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:15:22.957391 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:22.956869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:41583 (local address 127.0.0.1:59288)
0929 02:15:22.957087 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:22.957230 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260929 02:15:22.957724 6407 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
I20260929 02:15:22.959280 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:22.958941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:41583 (local address 127.0.0.1:59298)
0929 02:15:22.959086 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:22.959154 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 02:15:22.959592 6458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:41583: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.189.keytab.
Entry for principal kudu/127.24.216.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.189.keytab.
I20260929 02:15:22.981314 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.189:41583
--webserver_interface=127.24.216.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.189.keytab
--principal=kudu/127.24.216.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:40745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:45717,127.24.216.189:41583,127.24.216.188:45869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:15:23.008955 6458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } attempt: 1
I20260929 02:15:23.010485 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.009862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:41583 (local address 127.0.0.1:59312)
0929 02:15:23.010179 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.010311 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 02:15:23.011624 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.011221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:41583 (local address 127.0.0.1:59318)
0929 02:15:23.011377 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.011464 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 02:15:23.011822 6458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:41583: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
I20260929 02:15:23.080098 6458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } attempt: 2
I20260929 02:15:23.081517 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.080833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:41583 (local address 127.0.0.1:59326)
0929 02:15:23.081163 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.081298 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 02:15:23.082814 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.082324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:41583 (local address 127.0.0.1:59328)
0929 02:15:23.082516 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.082639 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:15:23.083050 6458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:41583: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
I20260929 02:15:23.174335 6458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } attempt: 3
I20260929 02:15:23.175727 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.175048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:41583 (local address 127.0.0.1:59344)
0929 02:15:23.175366 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.175525 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260929 02:15:23.177073 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.176551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:41583 (local address 127.0.0.1:59354)
0929 02:15:23.176750 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.176867 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 02:15:23.177342 6458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:41583: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
I20260929 02:15:23.320633 6458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } attempt: 4
I20260929 02:15:23.322000 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.321338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:41583 (local address 127.0.0.1:59364)
0929 02:15:23.321687 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.321807 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260929 02:15:23.323218 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.322811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:41583 (local address 127.0.0.1:59374)
0929 02:15:23.322984 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.323095 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260929 02:15:23.323567 6458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:41583: Network error: Client connection negotiation failed: client connection to 127.24.216.189:41583: connect: Connection refused (error 111)
W20260929 02:15:23.327356 6465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:23.327661 6465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:23.327821 6465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:23.335206 6465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:15:23.335309 6465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:23.335434 6465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:23.335530 6465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:15:23.335613 6465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:15:23.345610 6465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:40745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:45717,127.24.216.189:41583,127.24.216.188:45869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.189.keytab
--principal=kudu/127.24.216.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.189:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.216.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:23.347023 6465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:23.348388 6465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:23.360231 6470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:23.360404 6471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:23.360379 6473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:23.361258 6465 server_base.cc:1034] running on GCE node
I20260929 02:15:23.362010 6465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:23.363492 6465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:23.364689 6465 hybrid_clock.cc:648] HybridClock initialized: now 1790648123364628 us; error 70 us; skew 500 ppm
I20260929 02:15:23.365038 6465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:23 dist-test-slave-9k0h krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648123, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:23.368393 6465 init.cc:377] Logged in from keytab as kudu/127.24.216.189@KRBTEST.COM (short username kudu)
I20260929 02:15:23.370003 6465 webserver.cc:460] Webserver started at http://127.24.216.189:43767/ using document root <none> and password file <none>
I20260929 02:15:23.370775 6465 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:23.370925 6465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:23.371241 6465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:15:23.374450 6465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/data/instance:
uuid: "4220d2ceceda44b4b05caa4a31d34458"
format_stamp: "Formatted at 2026-09-29 02:15:23 on dist-test-slave-9k0h"
server_key: "40ed851b7089c308b15be26a4e38f93b"
server_key_iv: "920df5789e6e0a549f94f892fabd5ae8"
server_key_version: "encryptionkey@0"
I20260929 02:15:23.375284 6465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/wal/instance:
uuid: "4220d2ceceda44b4b05caa4a31d34458"
format_stamp: "Formatted at 2026-09-29 02:15:23 on dist-test-slave-9k0h"
server_key: "40ed851b7089c308b15be26a4e38f93b"
server_key_iv: "920df5789e6e0a549f94f892fabd5ae8"
server_key_version: "encryptionkey@0"
I20260929 02:15:23.380786 6465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:15:23.384402 6480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:23.385763 6465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:15:23.385942 6465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/wal
uuid: "4220d2ceceda44b4b05caa4a31d34458"
format_stamp: "Formatted at 2026-09-29 02:15:23 on dist-test-slave-9k0h"
server_key: "40ed851b7089c308b15be26a4e38f93b"
server_key_iv: "920df5789e6e0a549f94f892fabd5ae8"
server_key_version: "encryptionkey@0"
I20260929 02:15:23.386152 6465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:23.410382 6465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:23.414968 6465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:23.415277 6465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:23.434417 6465 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.189:41583
I20260929 02:15:23.434456 6531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.189:41583 every 8 connection(s)
I20260929 02:15:23.435822 6465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/data/info.pb
I20260929 02:15:23.436395 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 6465
I20260929 02:15:23.436605 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-1/wal/instance
I20260929 02:15:23.437321 25442 external_mini_cluster.cc:1428] Setting key 6ac7af315aa3e9229b71c8406412d311
I20260929 02:15:23.439924 6532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:23.446650 6532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.216.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.188@KRBTEST.COM" created.
Sep 29 02:15:23 dist-test-slave-9k0h krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648123, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.189@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:23.470496 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.449259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.24.216.190:45717)
0929 02:15:23.449490 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:23.449497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:23.450859 (+ 1362us) server_negotiation.cc:407] Connection header received
0929 02:15:23.452739 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:23.452748 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:23.452852 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:23.452994 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:23.454861 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.456093 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.457708 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.458427 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.458574 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.462679 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:23.462688 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:23.462696 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:23.462746 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:23.466725 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.467582 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.467590 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.467594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.467768 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.468286 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.468299 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.468304 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.468735 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:23.468894 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:23.469720 (+ 826us) server_negotiation.cc:299] Negotiation successful
0929 02:15:23.469993 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:15:23.471606 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.449819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:45717 (local address 127.0.0.1:47518)
0929 02:15:23.450671 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.450724 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:23.452440 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:23.453175 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:23.453183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:23.453940 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:23.454650 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:23.454670 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.456264 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.456267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:23.457546 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:23.457558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.458677 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.458684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:23.458882 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.459612 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:23.459650 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:23.462417 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:23.466899 (+ 4482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:23.466908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:23.466923 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:23.467415 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:23.467909 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:23.467912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:23.467915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:23.468130 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:23.468875 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:23.468881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:23.469276 (+ 395us) client_negotiation.cc:769] Sending connection context
0929 02:15:23.469547 (+ 271us) client_negotiation.cc:240] Negotiation successful
0929 02:15:23.469979 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":397,"threads_started":1}
I20260929 02:15:23.473388 6532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:15:23 dist-test-slave-9k0h krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648123, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.189@KRBTEST.COM for kudu/127.24.216.189@KRBTEST.COM
I20260929 02:15:23.488065 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.474210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:41583 (local address 127.0.0.1:59390)
0929 02:15:23.474395 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.474414 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:23.474510 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:23.475454 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:23.475458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:23.475876 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:23.475994 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:23.476002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.477299 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.477306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:23.478398 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:23.478409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.479276 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.479282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:23.479485 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.480130 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:23.480147 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:23.482227 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:23.485298 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:23.485304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:23.485308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:23.485760 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:23.486200 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:23.486205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:23.486209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:23.486359 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:23.487132 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:23.487136 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:23.487269 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 02:15:23.487413 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 02:15:23.487678 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 02:15:23.488376 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.474440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59390 (local address 127.24.216.189:41583)
0929 02:15:23.475071 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:23.475076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:23.475100 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:15:23.475141 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:23.475147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:23.475317 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:23.475468 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:23.476141 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.477171 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.478522 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.479139 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.479287 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.482418 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:23.482429 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:23.482442 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:23.482472 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:23.485162 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.485895 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.485901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.485903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.486008 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.486469 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.486474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.486478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.486996 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:23.487219 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:23.487588 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 02:15:23.487809 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":364,"threads_started":1}
I20260929 02:15:23.490974 6532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:15:23.492161 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.491705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:43926)
0929 02:15:23.491870 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.491995 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 02:15:23.492429 6484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.188.keytab.
Entry for principal kudu/127.24.216.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.188.keytab.
I20260929 02:15:23.494083 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.493672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:43938)
0929 02:15:23.493851 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.493941 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 02:15:23.494367 6532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:45869: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
I20260929 02:15:23.494583 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.188:45869
--webserver_interface=127.24.216.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.188.keytab
--principal=kudu/127.24.216.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:40745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:45717,127.24.216.189:41583,127.24.216.188:45869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:15:23.543692 6532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } attempt: 1
I20260929 02:15:23.544943 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.544493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:43942)
0929 02:15:23.544695 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.544798 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 02:15:23.545885 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.545551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:43948)
0929 02:15:23.545711 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.545782 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 02:15:23.546072 6532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:45869: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
I20260929 02:15:23.614276 6532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } attempt: 2
I20260929 02:15:23.615500 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.614946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:43964)
0929 02:15:23.615196 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.615310 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 02:15:23.616519 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.616150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:43972)
0929 02:15:23.616299 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.616382 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 02:15:23.616739 6532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:45869: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
I20260929 02:15:23.622824 6458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } attempt: 5
Sep 29 02:15:23 dist-test-slave-9k0h krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648122, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for kudu/127.24.216.189@KRBTEST.COM
I20260929 02:15:23.636041 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.623453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:41583 (local address 127.0.0.1:59398)
0929 02:15:23.623646 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.623667 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:23.623812 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:23.624164 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:23.624172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:23.624557 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:23.624735 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:23.624743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.625710 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.625716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:23.626673 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:23.626685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.627394 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.627400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:23.627579 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.628282 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:23.628301 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:23.630449 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:23.633512 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:23.633518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:23.633522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:23.633881 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:23.634316 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:23.634321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:23.634325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:23.634525 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:23.635068 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:23.635073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:23.635233 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 02:15:23.635379 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 02:15:23.635684 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 02:15:23.636045 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.623514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59398 (local address 127.24.216.189:41583)
0929 02:15:23.623680 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:23.623688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:23.623747 (+ 59us) server_negotiation.cc:407] Connection header received
0929 02:15:23.623937 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:23.623950 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:23.624022 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:23.624126 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:23.624885 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.625568 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.626812 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.627247 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.627381 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.630657 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:23.630662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:23.630665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:23.630698 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:23.633340 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.634012 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.634019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.634023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.634194 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.634636 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.634643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.634646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.634945 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:23.635105 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:23.635487 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 02:15:23.635726 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:15:23.637574 6458 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:15:23.638790 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.638327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:43976)
0929 02:15:23.638539 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.638631 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 02:15:23.639822 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.639428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:43984)
0929 02:15:23.639577 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.639687 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 02:15:23.640077 6458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:45869: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
I20260929 02:15:23.691323 6458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } attempt: 1
I20260929 02:15:23.692580 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.692000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:43988)
0929 02:15:23.692245 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.692374 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:15:23.693634 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.693260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:44002)
0929 02:15:23.693416 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.693504 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 02:15:23.693884 6458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:45869: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
I20260929 02:15:23.707958 6532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } attempt: 3
I20260929 02:15:23.709165 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.708600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:44016)
0929 02:15:23.708848 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.708959 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 02:15:23.710529 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.709923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:44018)
0929 02:15:23.710234 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.710348 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260929 02:15:23.710757 6532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:45869: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
I20260929 02:15:23.762108 6458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } attempt: 2
I20260929 02:15:23.763440 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.762819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:44032)
0929 02:15:23.763104 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.763224 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 02:15:23.764599 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.764142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:44042)
0929 02:15:23.764356 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.764450 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 02:15:23.764946 6458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:45869: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
W20260929 02:15:23.824218 6539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:23.824517 6539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:23.824618 6539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:23.831828 6539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:15:23.831943 6539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:23.831990 6539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:23.832064 6539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:15:23.832109 6539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:15:23.842041 6539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:40745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:45717,127.24.216.189:41583,127.24.216.188:45869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.188.keytab
--principal=kudu/127.24.216.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.188:45869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.216.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:23.843423 6539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:23.844672 6539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 02:15:23.853976 6532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } attempt: 4
I20260929 02:15:23.855358 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.854671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:44044)
0929 02:15:23.855030 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.855143 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
W20260929 02:15:23.856335 6544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:23.856400 6545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:23.856401 6547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:23.857138 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.856646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:44060)
0929 02:15:23.856920 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.857011 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260929 02:15:23.857364 6532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:45869: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
I20260929 02:15:23.857538 6539 server_base.cc:1034] running on GCE node
I20260929 02:15:23.858420 6539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:23.859915 6539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:23.861140 6539 hybrid_clock.cc:648] HybridClock initialized: now 1790648123861089 us; error 38 us; skew 500 ppm
I20260929 02:15:23.861491 6539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:23 dist-test-slave-9k0h krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648123, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:23.864571 6539 init.cc:377] Logged in from keytab as kudu/127.24.216.188@KRBTEST.COM (short username kudu)
I20260929 02:15:23.866045 6539 webserver.cc:460] Webserver started at http://127.24.216.188:35221/ using document root <none> and password file <none>
I20260929 02:15:23.866811 6539 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:23.866961 6539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:23.867268 6539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:15:23.870316 6539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/data/instance:
uuid: "cb35171939b5467a9a0426fd431c757f"
format_stamp: "Formatted at 2026-09-29 02:15:23 on dist-test-slave-9k0h"
server_key: "fc5ccd50979d979e3b4a8dee3473cc1f"
server_key_iv: "439a3dd4341159bdfa5a11d878227479"
server_key_version: "encryptionkey@0"
I20260929 02:15:23.871127 6539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/wal/instance:
uuid: "cb35171939b5467a9a0426fd431c757f"
format_stamp: "Formatted at 2026-09-29 02:15:23 on dist-test-slave-9k0h"
server_key: "fc5ccd50979d979e3b4a8dee3473cc1f"
server_key_iv: "439a3dd4341159bdfa5a11d878227479"
server_key_version: "encryptionkey@0"
I20260929 02:15:23.871160 6458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } attempt: 3
I20260929 02:15:23.872206 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.871744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:44064)
0929 02:15:23.871969 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.872046 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 02:15:23.873168 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.872789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:44068)
0929 02:15:23.872993 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.873062 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260929 02:15:23.873359 6458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:45869: Network error: Client connection negotiation failed: client connection to 127.24.216.188:45869: connect: Connection refused (error 111)
I20260929 02:15:23.876543 6539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 02:15:23.880411 6554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:23.881804 6539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:15:23.882009 6539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/wal
uuid: "cb35171939b5467a9a0426fd431c757f"
format_stamp: "Formatted at 2026-09-29 02:15:23 on dist-test-slave-9k0h"
server_key: "fc5ccd50979d979e3b4a8dee3473cc1f"
server_key_iv: "439a3dd4341159bdfa5a11d878227479"
server_key_version: "encryptionkey@0"
I20260929 02:15:23.882266 6539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:23.900472 6539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:23.904881 6539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:23.905184 6539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:23.925459 6539 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.188:45869
I20260929 02:15:23.925482 6605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.188:45869 every 8 connection(s)
I20260929 02:15:23.926891 6539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/data/info.pb
I20260929 02:15:23.928115 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 6539
I20260929 02:15:23.928308 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790647920678408-25442-0/minicluster-data/master-2/wal/instance
I20260929 02:15:23.928802 25442 external_mini_cluster.cc:1428] Setting key d676e77abdb7bdb41160a7c41e59e635
I20260929 02:15:23.931196 6606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:23.940128 6606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:15:23 dist-test-slave-9k0h krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.188@KRBTEST.COM
Sep 29 02:15:23 dist-test-slave-9k0h krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648123, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.188@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:23.953713 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.930738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.24.216.188:45869)
0929 02:15:23.931541 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:23.931556 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:23.931606 (+ 50us) server_negotiation.cc:407] Connection header received
0929 02:15:23.933908 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:23.933935 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:23.934470 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:23.934916 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:23.936239 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.938301 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.939565 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.940615 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.940801 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.945536 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:23.945547 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:23.945564 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:23.945604 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:23.948995 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.949886 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.949896 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.949906 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.950129 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.950594 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.950598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.950600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.951130 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:23.951353 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:23.951920 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 02:15:23.952407 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":346,"threads_started":1}
I20260929 02:15:23.959971 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.942968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:45717 (local address 127.0.0.1:47526)
0929 02:15:23.943814 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.943858 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:23.944051 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:23.944494 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:23.944501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:23.945281 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:23.945641 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:23.945656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.947060 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.947064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:23.948324 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:23.948337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.949377 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.949384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:23.949582 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.950365 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:23.950391 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:23.952954 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:23.957380 (+ 4426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:23.957388 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:23.957394 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:23.957918 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:23.958320 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:23.958325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:23.958328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:23.958512 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:23.959007 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:23.959015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:23.959243 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 02:15:23.959428 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 02:15:23.959679 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":409,"threads_started":1}
I20260929 02:15:23.960203 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.942989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.24.216.190:45717)
0929 02:15:23.943251 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:23.943259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:23.943962 (+ 703us) server_negotiation.cc:407] Connection header received
0929 02:15:23.944205 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:23.944210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:23.944299 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:23.944396 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:23.945800 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.946876 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.948476 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.949187 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.949289 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.953144 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:23.953149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:23.953153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:23.953200 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:23.957251 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.958034 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.958038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.958040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.958210 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.958632 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.958634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.958636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.958869 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:23.958978 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:23.959579 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 02:15:23.959825 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:15:23.961649 6606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:15:23 dist-test-slave-9k0h krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
Sep 29 02:15:23 dist-test-slave-9k0h krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648123, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.188@KRBTEST.COM for kudu/127.24.216.189@KRBTEST.COM
I20260929 02:15:23.973667 6611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.956828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.24.216.190:45717)
0929 02:15:23.957493 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:23.957496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:23.957514 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:15:23.957551 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:23.957554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:23.957609 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:23.957687 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:23.959316 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.960634 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.962283 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.962916 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.963042 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.967820 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:23.967824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:23.967828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:23.967856 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:23.970711 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.971596 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.971600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.971603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.971735 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.972196 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.972200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.972203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.972496 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:23.972775 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:23.973051 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 02:15:23.973297 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":305,"threads_started":1}
I20260929 02:15:23.975541 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.962337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:41583 (local address 127.0.0.1:59408)
0929 02:15:23.962503 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.962540 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:23.962644 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:23.962943 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:23.962950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:23.963360 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:23.963584 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:23.963608 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.964985 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.964988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:23.965827 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:23.965836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.966631 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.966637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:23.966863 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.967475 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:23.967499 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:23.969413 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:23.972726 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:23.972733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:23.972736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:23.973283 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:23.973758 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:23.973762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:23.973765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:23.973937 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:23.974642 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:23.974648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:23.974827 (+ 179us) client_negotiation.cc:769] Sending connection context
0929 02:15:23.974955 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 02:15:23.975237 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 02:15:23.975664 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.962410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.24.216.189:41583)
0929 02:15:23.962583 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:23.962588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:23.962604 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:15:23.962754 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:23.962758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:23.962816 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:23.962891 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:23.963762 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.964846 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.965967 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.966451 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.966602 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.969635 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:23.969641 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:23.969644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:23.969676 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:23.972522 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.973462 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.973468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.973472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.973634 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.974056 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.974061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.974064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.974505 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:23.974628 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:23.975078 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 02:15:23.975243 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:15:23.976970 6606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:15:23 dist-test-slave-9k0h krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648123, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.188@KRBTEST.COM for kudu/127.24.216.188@KRBTEST.COM
Sep 29 02:15:23 dist-test-slave-9k0h krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.189@KRBTEST.COM
I20260929 02:15:23.993983 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.977973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:44074)
0929 02:15:23.978272 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:23.978291 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:23.978389 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:23.978670 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:23.978674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:23.979094 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:23.979247 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:23.979255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.980247 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.980252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:23.981378 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:23.981392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.982574 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.982582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:23.982788 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.983390 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:23.983419 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:23.986246 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:23.991102 (+ 4856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:23.991108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:23.991111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:23.991566 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:23.992027 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:23.992034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:23.992037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:23.992204 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:23.992917 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:23.992921 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:23.993074 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 02:15:23.993255 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 02:15:23.993549 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260929 02:15:23.994246 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.977894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44074 (local address 127.24.216.188:45869)
0929 02:15:23.978052 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:23.978059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:23.978340 (+ 281us) server_negotiation.cc:407] Connection header received
0929 02:15:23.978489 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:23.978501 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:23.978560 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:23.978656 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:23.979431 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.980110 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.981536 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.982312 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.982423 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.986526 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:23.986535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:23.986540 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:23.986587 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:23.990937 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.991714 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.991721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.991724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.991884 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.992385 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.992391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.992394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.992799 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:23.992908 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:23.993393 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 02:15:23.993766 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":56}
I20260929 02:15:23.994552 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:23.975735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.24.216.189:41583)
0929 02:15:23.976397 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:23.976401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:23.976421 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:15:23.976461 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:23.976464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:23.976515 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:23.976596 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:23.978321 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.979246 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.980612 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:23.981025 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:23.981117 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:23.987663 (+ 6546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:23.987673 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:23.987680 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:23.987729 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:23.991137 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.992062 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.992069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.992075 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.992266 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:23.992701 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:23.992704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:23.992706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:23.993068 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:23.993363 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:23.993834 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 02:15:23.994114 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":340,"threads_started":1}
I20260929 02:15:23.999820 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:15:24.000563 6606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f: Bootstrap starting.
I20260929 02:15:24.005350 6606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:24.007125 6606 log.cc:824] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:24.017163 6606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f: No bootstrap required, opened a new log
I20260929 02:15:24.029768 6611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:24.006054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.24.216.190:45717)
0929 02:15:24.006312 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:24.006321 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:24.006766 (+ 445us) server_negotiation.cc:407] Connection header received
0929 02:15:24.007188 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:24.007194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:24.007290 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:24.007376 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:24.010913 (+ 3537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:24.012382 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:24.015199 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:24.016125 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:24.016259 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:24.022183 (+ 5924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:24.022191 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:24.022195 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:24.022241 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:24.026138 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:24.027270 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:24.027276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:24.027280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:24.027427 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:24.027889 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:24.027895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:24.027901 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:24.028287 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:24.028506 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:24.029098 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 02:15:24.029344 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:15:24.029448 6606 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } }
I20260929 02:15:24.029944 6606 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:24.030089 6606 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb35171939b5467a9a0426fd431c757f, State: Initialized, Role: FOLLOWER
I20260929 02:15:24.030972 6606 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } }
I20260929 02:15:24.032100 6621 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } } }
I20260929 02:15:24.032661 6621 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:15:24.033512 6606 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:15:24.035804 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:24.007420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.24.216.188:45869)
0929 02:15:24.007607 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:24.007615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:24.015676 (+ 8061us) server_negotiation.cc:407] Connection header received
0929 02:15:24.016172 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:24.016178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:24.016259 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:24.016329 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:24.018959 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:24.020133 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:24.022454 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:24.023255 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:24.023400 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:24.027338 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:24.027347 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:24.027356 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:24.027399 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:24.031032 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:24.033608 (+ 2576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:24.033616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:24.033622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:24.033800 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:24.034324 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:24.034331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:24.034336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:24.034690 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:24.034974 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:24.035288 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 02:15:24.035528 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:15:24.039160 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:24.006976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.24.216.189:41583)
0929 02:15:24.010640 (+ 3664us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:24.010650 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:24.022189 (+ 11539us) server_negotiation.cc:407] Connection header received
0929 02:15:24.022489 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:24.022497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:24.022584 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:24.022655 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:24.024904 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:24.026005 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:24.028139 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:24.028774 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:24.028877 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:24.032559 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:24.032565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:24.032570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:24.032615 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:24.036408 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:24.037082 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:24.037088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:24.037091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:24.037221 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:24.037612 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:24.037617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:24.037621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:24.037958 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:24.038225 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:24.038572 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 02:15:24.038798 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3469}
W20260929 02:15:24.049976 6632 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:15:24.050112 6632 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:15:24.050552 6458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } attempt: 4
Sep 29 02:15:24 dist-test-slave-9k0h krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648122, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for kudu/127.24.216.188@KRBTEST.COM
I20260929 02:15:24.046782 6539 master_runner.cc:186] Error getting master registration for 127.24.216.190:45717: OK, instance_id { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" instance_seqno: 1790648122907978 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:15:24.060365 6539 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:15:24.067538 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:24.051091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:44094)
0929 02:15:24.051335 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:24.051355 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:24.051461 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:24.051856 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:24.051861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:24.052392 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:24.052636 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:24.052648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:24.054363 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:24.054369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:24.055834 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:24.055850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:24.057087 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:24.057094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:24.057282 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:24.057969 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:24.057994 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:24.060791 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:24.064719 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:24.064726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:24.064730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:24.065344 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:24.065822 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:24.065826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:24.065830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:24.066002 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:24.066766 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:24.066771 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:24.066946 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 02:15:24.067068 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 02:15:24.067275 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260929 02:15:24.067879 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:24.051250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.24.216.188:45869)
0929 02:15:24.051421 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:24.051427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:24.051445 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:15:24.051599 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:24.051604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:24.051696 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:24.051812 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:24.052793 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:24.054134 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:24.055986 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:24.056735 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:24.056880 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:24.060966 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:24.060974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:24.060978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:24.061018 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:24.064534 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:24.065483 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:24.065488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:24.065493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:24.065673 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:24.066143 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:24.066182 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:24.066186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:24.066602 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:24.066746 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:24.067214 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 02:15:24.067454 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:15:24.072208 6458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33: Bootstrap starting.
I20260929 02:15:24.075467 6458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:24.077256 6458 log.cc:824] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:24.080277 6458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33: No bootstrap required, opened a new log
I20260929 02:15:24.090569 6458 raft_consensus.cc:348] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } }
I20260929 02:15:24.090996 6458 raft_consensus.cc:374] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:24.091135 6458 raft_consensus.cc:729] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a73e32944be943b1849653bc54ba1b33, State: Initialized, Role: FOLLOWER
I20260929 02:15:24.091843 6458 consensus_queue.cc:260] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } }
I20260929 02:15:24.093779 6635 sys_catalog.cc:455] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } } }
I20260929 02:15:24.094125 6635 sys_catalog.cc:458] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:15:24.095311 6458 sys_catalog.cc:564] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:15:24.105309 6646 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:15:24.105480 6646 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:15:24.105891 6391 master_runner.cc:186] Error getting master registration for 127.24.216.189:41583: OK, instance_id { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" instance_seqno: 1790648123428944 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:15:24.113498 6391 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 02:15:24.147538 6412 tablet.cc:2369] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:15:24.156653 6532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } attempt: 5
W20260929 02:15:24.157545 6560 tablet.cc:2369] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:15:24 dist-test-slave-9k0h krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648123, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.189@KRBTEST.COM for kudu/127.24.216.188@KRBTEST.COM
I20260929 02:15:24.171953 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:24.157315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:45869 (local address 127.0.0.1:44098)
0929 02:15:24.157606 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:24.157629 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:24.157750 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:24.159211 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:24.159215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:24.159796 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:24.159996 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:24.160007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:24.161470 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:24.161478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:24.162538 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:24.162550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:24.163291 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:24.163297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:24.163532 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:24.164150 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:24.164167 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:24.166133 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:24.169417 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:24.169423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:24.169427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:24.169818 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:24.170206 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:24.170211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:24.170214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:24.170380 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:24.170936 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:24.170940 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:24.171098 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 02:15:24.171247 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 02:15:24.171557 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260929 02:15:24.172406 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:24.158566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44098 (local address 127.24.216.188:45869)
0929 02:15:24.158763 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:24.158769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:24.158787 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:15:24.158845 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:24.158853 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:24.158934 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:24.159031 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:24.160169 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:24.161312 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:24.162686 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:24.163141 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:24.163277 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:24.166355 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:24.166360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:24.166363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:24.166392 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:24.169280 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:24.169935 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:24.169943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:24.169947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:24.170071 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:24.170497 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:24.170506 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:24.170509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:24.170805 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:24.170976 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:24.171460 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 02:15:24.171709 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 02:15:24.176838 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458: Bootstrap starting.
I20260929 02:15:24.180997 6532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:24.182381 6532 log.cc:824] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:24.185103 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458: No bootstrap required, opened a new log
I20260929 02:15:24.193495 6532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } }
I20260929 02:15:24.193861 6532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:24.194005 6532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4220d2ceceda44b4b05caa4a31d34458, State: Initialized, Role: FOLLOWER
I20260929 02:15:24.194693 6532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } }
I20260929 02:15:24.195657 6649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } } }
I20260929 02:15:24.196112 6649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:15:24.197178 6532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:15:24.221086 6635 raft_consensus.cc:482] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:15:24.221426 6635 raft_consensus.cc:504] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } }
I20260929 02:15:24.223472 6635 leader_election.cc:288] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4220d2ceceda44b4b05caa4a31d34458 (127.24.216.189:41583), cb35171939b5467a9a0426fd431c757f (127.24.216.188:45869)
I20260929 02:15:24.224925 6581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a73e32944be943b1849653bc54ba1b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb35171939b5467a9a0426fd431c757f" is_pre_election: true
I20260929 02:15:24.225697 6581 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a73e32944be943b1849653bc54ba1b33 in term 0.
I20260929 02:15:24.226481 6410 leader_election.cc:302] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a73e32944be943b1849653bc54ba1b33, cb35171939b5467a9a0426fd431c757f; no voters:
I20260929 02:15:24.228884 6507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a73e32944be943b1849653bc54ba1b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4220d2ceceda44b4b05caa4a31d34458" is_pre_election: true
I20260929 02:15:24.229543 6507 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a73e32944be943b1849653bc54ba1b33 in term 0.
I20260929 02:15:24.232713 6465 master_runner.cc:263] No leader master found from master 4220d2ceceda44b4b05caa4a31d34458
I20260929 02:15:24.232864 6465 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:15:24.233052 6635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:15:24.233230 6635 raft_consensus.cc:482] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:15:24.233423 6635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:24.238090 6635 raft_consensus.cc:504] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } }
I20260929 02:15:24.239451 6507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a73e32944be943b1849653bc54ba1b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4220d2ceceda44b4b05caa4a31d34458"
I20260929 02:15:24.239748 6507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:24.244486 6507 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a73e32944be943b1849653bc54ba1b33 in term 1.
I20260929 02:15:24.244956 6407 leader_election.cc:302] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4220d2ceceda44b4b05caa4a31d34458, a73e32944be943b1849653bc54ba1b33; no voters:
I20260929 02:15:24.253902 6581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a73e32944be943b1849653bc54ba1b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb35171939b5467a9a0426fd431c757f"
I20260929 02:15:24.254216 6581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:24.258800 6581 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a73e32944be943b1849653bc54ba1b33 in term 1.
I20260929 02:15:24.259722 6635 leader_election.cc:288] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [CANDIDATE]: Term 1 election: Requested vote from peers 4220d2ceceda44b4b05caa4a31d34458 (127.24.216.189:41583), cb35171939b5467a9a0426fd431c757f (127.24.216.188:45869)
I20260929 02:15:24.260048 6635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:15:24.260247 6635 raft_consensus.cc:686] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [term 1 LEADER]: Becoming Leader. State: Replica: a73e32944be943b1849653bc54ba1b33, State: Running, Role: LEADER
I20260929 02:15:24.261021 6635 consensus_queue.cc:237] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } }
I20260929 02:15:24.280086 6662 sys_catalog.cc:455] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a73e32944be943b1849653bc54ba1b33. Latest consensus state: current_term: 1 leader_uuid: "a73e32944be943b1849653bc54ba1b33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } } }
I20260929 02:15:24.280675 6662 sys_catalog.cc:458] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [sys.catalog]: This master's current role is: LEADER
W20260929 02:15:24.284181 6660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:15:24.284406 6660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:15:24.285293 6664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:15:24.287843 6664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:15:24.305403 6507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [term 1 FOLLOWER]: Refusing update from remote peer a73e32944be943b1849653bc54ba1b33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:15:24.307132 6662 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:15:24.308468 6581 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [term 1 FOLLOWER]: Refusing update from remote peer a73e32944be943b1849653bc54ba1b33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:15:24.309813 6662 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:15:24.333125 6649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a73e32944be943b1849653bc54ba1b33. Latest consensus state: current_term: 1 leader_uuid: "a73e32944be943b1849653bc54ba1b33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } } }
I20260929 02:15:24.333460 6649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:15:24.342945 6621 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a73e32944be943b1849653bc54ba1b33. Latest consensus state: current_term: 1 leader_uuid: "a73e32944be943b1849653bc54ba1b33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } } }
I20260929 02:15:24.343263 6621 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:15:24.345568 6665 mvcc.cc:205] Tried to move back new op lower bound from 7334494717141553152 to 7334494716991410176. Current Snapshot: MvccSnapshot[applied={T|T < 7334494717141553152}]
I20260929 02:15:24.373409 6649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a73e32944be943b1849653bc54ba1b33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } } }
I20260929 02:15:24.373706 6649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:15:24.380371 6664 catalog_manager.cc:1347] Generated new cluster ID: 1a95c920f8fd4f1d8079c7080918c93a
I20260929 02:15:24.380504 6664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:15:24.383189 6635 sys_catalog.cc:455] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a73e32944be943b1849653bc54ba1b33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } } }
I20260929 02:15:24.383491 6635 sys_catalog.cc:458] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:24.392845 6662 sys_catalog.cc:455] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a73e32944be943b1849653bc54ba1b33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } } }
I20260929 02:15:24.394622 6662 sys_catalog.cc:458] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33 [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:24.415220 6621 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a73e32944be943b1849653bc54ba1b33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a73e32944be943b1849653bc54ba1b33" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45717 } } peers { permanent_uuid: "4220d2ceceda44b4b05caa4a31d34458" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 41583 } } peers { permanent_uuid: "cb35171939b5467a9a0426fd431c757f" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 45869 } } }
I20260929 02:15:24.415504 6621 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:15:24.421898 6664 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:15:24.423821 6664 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:15:24.448185 6664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a73e32944be943b1849653bc54ba1b33: Generated new TSK 0
I20260929 02:15:24.449275 6664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:15:25.052055 6632 catalog_manager.cc:1259] Loaded cluster ID: 1a95c920f8fd4f1d8079c7080918c93a
I20260929 02:15:25.052168 6632 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f: loading cluster ID for follower catalog manager: success
I20260929 02:15:25.054613 6632 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f: acquiring CA information for follower catalog manager: success
I20260929 02:15:25.056368 6632 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cb35171939b5467a9a0426fd431c757f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:15:25.287338 6660 catalog_manager.cc:1259] Loaded cluster ID: 1a95c920f8fd4f1d8079c7080918c93a
I20260929 02:15:25.287490 6660 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458: loading cluster ID for follower catalog manager: success
I20260929 02:15:25.291898 6660 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458: acquiring CA information for follower catalog manager: success
I20260929 02:15:25.294833 6660 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4220d2ceceda44b4b05caa4a31d34458: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:15:25.606716 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 6391
I20260929 02:15:25.607955 6453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:25.720953 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 6391
W20260929 02:15:25.743798 6557 connection.cc:535] server connection from 127.0.0.1:44094 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 02:15:25.744189 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 6465
I20260929 02:15:25.745342 6527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:25.853377 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 6465
I20260929 02:15:25.873878 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 6539
I20260929 02:15:25.875299 6601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:25.984485 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 6539
2026-09-29T02:15:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4918 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:15:26 dist-test-slave-9k0h krb5kdc[6684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:15:26 dist-test-slave-9k0h krb5kdc[6684](info): set up 2 sockets
Sep 29 02:15:26 dist-test-slave-9k0h krb5kdc[6684](info): commencing operation
krb5kdc: starting...
W20260929 02:15:27.102756 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.006s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:15:27 dist-test-slave-9k0h krb5kdc[6684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:15:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:15:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:15:27.367031 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:35323
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:35323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:27.705164 6697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:27.705550 6697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:27.705724 6697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:27.713145 6697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:15:27.713320 6697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:27.713441 6697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:27.713550 6697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:15:27.713618 6697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:15:27.723798 6697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:35323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:35323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:27.725307 6697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:27.726817 6697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:27.738904 6705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:27.739161 6703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:27.739212 6702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:27.739923 6697 server_base.cc:1034] running on GCE node
I20260929 02:15:27.740706 6697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:27.742094 6697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:27.743343 6697 hybrid_clock.cc:648] HybridClock initialized: now 1790648127743297 us; error 32 us; skew 500 ppm
I20260929 02:15:27.743719 6697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:27 dist-test-slave-9k0h krb5kdc[6684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648127, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:27.753223 6697 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:15:27.754841 6697 webserver.cc:460] Webserver started at http://127.24.216.190:45101/ using document root <none> and password file <none>
I20260929 02:15:27.755499 6697 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:27.755609 6697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:27.755861 6697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:15:27.758879 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "47d85c496c0f43fbb0bcba6e5f7fae02"
format_stamp: "Formatted at 2026-09-29 02:15:27 on dist-test-slave-9k0h"
server_key: "a6d38d53acad991b8c247b8f6f330a8b"
server_key_iv: "604e370054254f4a370da02b34f9575e"
server_key_version: "encryptionkey@0"
I20260929 02:15:27.759593 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "47d85c496c0f43fbb0bcba6e5f7fae02"
format_stamp: "Formatted at 2026-09-29 02:15:27 on dist-test-slave-9k0h"
server_key: "a6d38d53acad991b8c247b8f6f330a8b"
server_key_iv: "604e370054254f4a370da02b34f9575e"
server_key_version: "encryptionkey@0"
I20260929 02:15:27.764829 6697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:15:27.768430 6712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:27.769842 6697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:15:27.770016 6697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "47d85c496c0f43fbb0bcba6e5f7fae02"
format_stamp: "Formatted at 2026-09-29 02:15:27 on dist-test-slave-9k0h"
server_key: "a6d38d53acad991b8c247b8f6f330a8b"
server_key_iv: "604e370054254f4a370da02b34f9575e"
server_key_version: "encryptionkey@0"
I20260929 02:15:27.770268 6697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:27.790097 6697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:27.794476 6697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:27.794775 6697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:27.813807 6697 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:35323
I20260929 02:15:27.813836 6763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:35323 every 8 connection(s)
I20260929 02:15:27.815394 6697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:15:27.819285 6764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:27.822964 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 6697
I20260929 02:15:27.823220 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:15:27.824061 25442 external_mini_cluster.cc:1428] Setting key 8cf9a7798687b331a60e51a5451920a1
I20260929 02:15:27.833091 6764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02: Bootstrap starting.
I20260929 02:15:27.837056 6764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:27.838568 6764 log.cc:824] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:15:27 dist-test-slave-9k0h krb5kdc[6684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:27.842053 6764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02: No bootstrap required, opened a new log
I20260929 02:15:27.848517 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:27.826021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.24.216.190:35323)
0929 02:15:27.826913 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:27.826924 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:27.826977 (+ 53us) server_negotiation.cc:407] Connection header received
0929 02:15:27.828728 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:27.828757 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:27.829212 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:27.829563 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:27.831527 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:27.832895 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:27.834788 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:27.835467 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:27.835604 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:27.841054 (+ 5450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:27.841061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:27.841076 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:27.841122 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:27.844333 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:27.845207 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:27.845212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:27.845219 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:27.845388 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:27.845782 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:27.845784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:27.845786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:27.846381 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:27.846612 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:27.847109 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 02:15:27.847475 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":388,"threads_started":1}
I20260929 02:15:27.851151 6764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d85c496c0f43fbb0bcba6e5f7fae02" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35323 } }
I20260929 02:15:27.851624 6764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:27.851763 6764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d85c496c0f43fbb0bcba6e5f7fae02, State: Initialized, Role: FOLLOWER
I20260929 02:15:27.852546 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:15:27.852452 6764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d85c496c0f43fbb0bcba6e5f7fae02" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35323 } }
I20260929 02:15:27.852738 6764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:15:27.852859 6764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:15:27.853024 6764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:27.856179 6764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d85c496c0f43fbb0bcba6e5f7fae02" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35323 } }
I20260929 02:15:27.856773 6764 leader_election.cc:302] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47d85c496c0f43fbb0bcba6e5f7fae02; no voters:
I20260929 02:15:27.857327 6764 leader_election.cc:288] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:15:27.857477 6769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:15:27.858199 6769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [term 1 LEADER]: Becoming Leader. State: Replica: 47d85c496c0f43fbb0bcba6e5f7fae02, State: Running, Role: LEADER
I20260929 02:15:27.858687 6769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d85c496c0f43fbb0bcba6e5f7fae02" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35323 } }
I20260929 02:15:27.859470 6764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:15:27.861577 6771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47d85c496c0f43fbb0bcba6e5f7fae02. Latest consensus state: current_term: 1 leader_uuid: "47d85c496c0f43fbb0bcba6e5f7fae02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d85c496c0f43fbb0bcba6e5f7fae02" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35323 } } }
I20260929 02:15:27.861564 6770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47d85c496c0f43fbb0bcba6e5f7fae02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d85c496c0f43fbb0bcba6e5f7fae02" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35323 } } }
I20260929 02:15:27.862015 6771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:27.862026 6770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02 [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:27.863531 6774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:15:27.868489 6774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:15:27.877899 6774 catalog_manager.cc:1347] Generated new cluster ID: 71dc8b515401415c8166033f8d01c7e0
I20260929 02:15:27.878026 6774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:15:27.901412 6774 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:15:27.902899 6774 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:15:27.912308 6774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47d85c496c0f43fbb0bcba6e5f7fae02: Generated new TSK 0
I20260929 02:15:27.913209 6774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:15:27.966266 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 6697
I20260929 02:15:27.967541 6759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:28.066644 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 6697
2026-09-29T02:15:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2079 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:15:28 dist-test-slave-9k0h krb5kdc[6792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:15:28 dist-test-slave-9k0h krb5kdc[6792](info): set up 2 sockets
Sep 29 02:15:28 dist-test-slave-9k0h krb5kdc[6792](info): commencing operation
krb5kdc: starting...
W20260929 02:15:29.230600 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:15:29 dist-test-slave-9k0h krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:15:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:15:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:15:29.493422 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:43547
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:43547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:29.823964 6805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:29.824271 6805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:29.824421 6805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:29.831808 6805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:15:29.831918 6805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:29.831995 6805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:29.832079 6805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:15:29.832162 6805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:15:29.842398 6805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:43547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:43547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:29.843892 6805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:29.845449 6805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:29.858225 6811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:29.858254 6813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:29.858289 6810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:29.859012 6805 server_base.cc:1034] running on GCE node
I20260929 02:15:29.859844 6805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:29.861318 6805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:29.862553 6805 hybrid_clock.cc:648] HybridClock initialized: now 1790648129862518 us; error 62 us; skew 500 ppm
I20260929 02:15:29.862936 6805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:29 dist-test-slave-9k0h krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648129, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:29.866287 6805 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:15:29.867942 6805 webserver.cc:460] Webserver started at http://127.24.216.190:36887/ using document root <none> and password file <none>
I20260929 02:15:29.868651 6805 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:29.868793 6805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:29.869109 6805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:15:29.872131 6805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "f367efcbd04c450fa43cff72647fa490"
format_stamp: "Formatted at 2026-09-29 02:15:29 on dist-test-slave-9k0h"
server_key: "0aca06459ae82803e4f84e449ecea49c"
server_key_iv: "a47dc546e3fd908026c698649c357594"
server_key_version: "encryptionkey@0"
I20260929 02:15:29.872964 6805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "f367efcbd04c450fa43cff72647fa490"
format_stamp: "Formatted at 2026-09-29 02:15:29 on dist-test-slave-9k0h"
server_key: "0aca06459ae82803e4f84e449ecea49c"
server_key_iv: "a47dc546e3fd908026c698649c357594"
server_key_version: "encryptionkey@0"
I20260929 02:15:29.878414 6805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:15:29.881990 6820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:29.883466 6805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:15:29.883656 6805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "f367efcbd04c450fa43cff72647fa490"
format_stamp: "Formatted at 2026-09-29 02:15:29 on dist-test-slave-9k0h"
server_key: "0aca06459ae82803e4f84e449ecea49c"
server_key_iv: "a47dc546e3fd908026c698649c357594"
server_key_version: "encryptionkey@0"
I20260929 02:15:29.883855 6805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:29.897753 6805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:29.902194 6805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:29.902508 6805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:29.921351 6805 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:43547
I20260929 02:15:29.921382 6871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:43547 every 8 connection(s)
I20260929 02:15:29.922802 6805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:15:29.926628 6872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:29.929054 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 6805
I20260929 02:15:29.929294 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:15:29.929919 25442 external_mini_cluster.cc:1428] Setting key 20e02c6fb0c20229ced2646eb4e48eb6
I20260929 02:15:29.939859 6872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Bootstrap starting.
I20260929 02:15:29.943575 6872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:29.944646 6872 log.cc:824] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:15:29 dist-test-slave-9k0h krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:29.948011 6872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: No bootstrap required, opened a new log
I20260929 02:15:29.955227 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:29.932081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.24.216.190:43547)
0929 02:15:29.932862 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:29.932878 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:29.932928 (+ 50us) server_negotiation.cc:407] Connection header received
0929 02:15:29.935447 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:29.935490 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:29.936056 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:29.936536 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:29.937852 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:29.939911 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:29.941418 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:29.942552 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:29.942746 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:29.947197 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:29.947208 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:29.947227 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:29.947270 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:29.950791 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:29.951543 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:29.951550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:29.951557 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:29.951720 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:29.952116 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:29.952125 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:29.952133 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:29.952744 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:29.953085 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:29.953545 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 02:15:29.953991 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":316,"threads_started":1}
I20260929 02:15:29.956137 6872 raft_consensus.cc:348] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } }
I20260929 02:15:29.956544 6872 raft_consensus.cc:374] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:29.956677 6872 raft_consensus.cc:729] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f367efcbd04c450fa43cff72647fa490, State: Initialized, Role: FOLLOWER
I20260929 02:15:29.957250 6872 consensus_queue.cc:260] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } }
I20260929 02:15:29.957512 6872 raft_consensus.cc:388] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:15:29.957664 6872 raft_consensus.cc:482] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:15:29.957854 6872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:29.960307 6872 raft_consensus.cc:504] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } }
I20260929 02:15:29.961114 6872 leader_election.cc:302] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f367efcbd04c450fa43cff72647fa490; no voters:
I20260929 02:15:29.961810 6872 leader_election.cc:288] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:15:29.961915 6877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:15:29.963591 6877 raft_consensus.cc:686] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 1 LEADER]: Becoming Leader. State: Replica: f367efcbd04c450fa43cff72647fa490, State: Running, Role: LEADER
I20260929 02:15:29.964210 6877 consensus_queue.cc:237] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } }
I20260929 02:15:29.964468 6872 sys_catalog.cc:564] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:15:29.967063 6879 sys_catalog.cc:455] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f367efcbd04c450fa43cff72647fa490. Latest consensus state: current_term: 1 leader_uuid: "f367efcbd04c450fa43cff72647fa490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } } }
I20260929 02:15:29.967154 6878 sys_catalog.cc:455] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f367efcbd04c450fa43cff72647fa490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } } }
I20260929 02:15:29.967510 6879 sys_catalog.cc:458] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:29.967567 6878 sys_catalog.cc:458] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:29.970686 6884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:15:29.976753 6884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:15:29.987722 6884 catalog_manager.cc:1347] Generated new cluster ID: 140f8011371446029a742427cdf3ceca
I20260929 02:15:29.987895 6884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:15:30.008538 6884 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:15:30.010592 6884 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:15:30.029060 6884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Generated new TSK 0
I20260929 02:15:30.030254 6884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:15:30.107304 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43547
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:30.475001 6898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:30.475293 6898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:30.475454 6898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:30.482945 6898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:30.483076 6898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:30.483238 6898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:15:30.493460 6898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:30.494916 6898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:30.496255 6898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:30.509938 6903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:30.509989 6906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:30.510015 6904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:30.510773 6898 server_base.cc:1034] running on GCE node
I20260929 02:15:30.511474 6898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:30.512614 6898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:30.513855 6898 hybrid_clock.cc:648] HybridClock initialized: now 1790648130513803 us; error 49 us; skew 500 ppm
I20260929 02:15:30.514273 6898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:30 dist-test-slave-9k0h krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648130, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:30.517880 6898 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:15:30.519666 6898 webserver.cc:460] Webserver started at http://127.24.216.129:39471/ using document root <none> and password file <none>
I20260929 02:15:30.520454 6898 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:30.520617 6898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:30.520946 6898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:15:30.524310 6898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "f865c9a4a7954672995d27c9079234b0"
format_stamp: "Formatted at 2026-09-29 02:15:30 on dist-test-slave-9k0h"
server_key: "221133f7e657e03c4ce73c935caf98ed"
server_key_iv: "69d4708388696d2fa856fb18ca77e855"
server_key_version: "encryptionkey@0"
I20260929 02:15:30.525233 6898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "f865c9a4a7954672995d27c9079234b0"
format_stamp: "Formatted at 2026-09-29 02:15:30 on dist-test-slave-9k0h"
server_key: "221133f7e657e03c4ce73c935caf98ed"
server_key_iv: "69d4708388696d2fa856fb18ca77e855"
server_key_version: "encryptionkey@0"
I20260929 02:15:30.531121 6898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:15:30.534953 6913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:30.536429 6898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:15:30.536576 6898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "f865c9a4a7954672995d27c9079234b0"
format_stamp: "Formatted at 2026-09-29 02:15:30 on dist-test-slave-9k0h"
server_key: "221133f7e657e03c4ce73c935caf98ed"
server_key_iv: "69d4708388696d2fa856fb18ca77e855"
server_key_version: "encryptionkey@0"
I20260929 02:15:30.536755 6898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:30.553706 6898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:30.557940 6898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:30.558256 6898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:30.559162 6898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:30.560534 6898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:30.560607 6898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:30.560694 6898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:30.560737 6898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:30.593926 6898 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:42215
I20260929 02:15:30.593969 7025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:42215 every 8 connection(s)
I20260929 02:15:30.595254 6898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:15:30.596551 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 6898
I20260929 02:15:30.596786 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:15:30.597581 25442 external_mini_cluster.cc:1428] Setting key 083b19ddcc7dca1666cd16b97685b2c7
Sep 29 02:15:30 dist-test-slave-9k0h krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648130, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:30.617960 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:30.598102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:33139 (local address 127.24.216.190:43547)
0929 02:15:30.598654 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:30.598658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:30.599890 (+ 1232us) server_negotiation.cc:407] Connection header received
0929 02:15:30.601646 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:30.601651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:30.601743 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:30.601862 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:30.603511 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:30.604252 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:30.606280 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:30.606995 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:30.607165 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:30.611580 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:30.611586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:30.611589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:30.611623 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:30.614379 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:30.615464 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:30.615479 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:30.615481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:30.615612 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:30.616181 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:30.616185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:30.616187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:30.616458 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:30.616576 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:30.617518 (+ 942us) server_negotiation.cc:299] Negotiation successful
0929 02:15:30.617704 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":266,"threads_started":1}
I20260929 02:15:30.619690 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:30.598728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:33139)
0929 02:15:30.599622 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:30.599688 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:30.601374 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:30.602105 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:30.602114 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:30.602771 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:30.603346 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:30.603363 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:30.604409 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:30.604419 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:30.606092 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:30.606108 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:30.607227 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:30.607235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:30.607536 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:30.608307 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:30.608331 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:30.611272 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:30.614546 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:30.614560 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:30.614595 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:30.615308 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:30.615734 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:30.615741 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:30.615748 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:30.616018 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:30.616604 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:30.616615 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:30.617100 (+ 485us) client_negotiation.cc:769] Sending connection context
0929 02:15:30.617425 (+ 325us) client_negotiation.cc:240] Negotiation successful
0929 02:15:30.618071 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":322,"threads_started":1}
I20260929 02:15:30.621528 7026 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43547
I20260929 02:15:30.621860 7026 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:30.622656 7026 heartbeater.cc:511] Master 127.24.216.190:43547 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
I20260929 02:15:30.624819 6837 ts_manager.cc:194] Registered new tserver with Master: f865c9a4a7954672995d27c9079234b0 (127.24.216.129:42215)
I20260929 02:15:30.626345 6837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:33139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:15:30.658363 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43547
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:31.013031 7032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:31.013379 7032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:31.013556 7032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:31.021669 7032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:31.021867 7032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:31.022054 7032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:15:31.032447 7032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:31.033824 7032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:31.035382 7032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:31.049176 7037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:31.049227 7038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:31.049540 7040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:31.050225 7032 server_base.cc:1034] running on GCE node
I20260929 02:15:31.050884 7032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:31.051874 7032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:31.053097 7032 hybrid_clock.cc:648] HybridClock initialized: now 1790648131053034 us; error 43 us; skew 500 ppm
I20260929 02:15:31.053421 7032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:31 dist-test-slave-9k0h krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648131, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:31.056751 7032 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:15:31.058437 7032 webserver.cc:460] Webserver started at http://127.24.216.130:46683/ using document root <none> and password file <none>
I20260929 02:15:31.059217 7032 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:31.059348 7032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:31.059674 7032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:15:31.062779 7032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "a14fe4898e104f168d39c7f865cc170e"
format_stamp: "Formatted at 2026-09-29 02:15:31 on dist-test-slave-9k0h"
server_key: "56515a29b988783036896b3928731b6a"
server_key_iv: "375ffab8507042a2806a62f89587d5ac"
server_key_version: "encryptionkey@0"
I20260929 02:15:31.063593 7032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "a14fe4898e104f168d39c7f865cc170e"
format_stamp: "Formatted at 2026-09-29 02:15:31 on dist-test-slave-9k0h"
server_key: "56515a29b988783036896b3928731b6a"
server_key_iv: "375ffab8507042a2806a62f89587d5ac"
server_key_version: "encryptionkey@0"
I20260929 02:15:31.069036 7032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:15:31.072986 7047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:31.074548 7032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:15:31.074731 7032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "a14fe4898e104f168d39c7f865cc170e"
format_stamp: "Formatted at 2026-09-29 02:15:31 on dist-test-slave-9k0h"
server_key: "56515a29b988783036896b3928731b6a"
server_key_iv: "375ffab8507042a2806a62f89587d5ac"
server_key_version: "encryptionkey@0"
I20260929 02:15:31.074950 7032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:31.101790 7032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:31.106312 7032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:31.106639 7032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:31.107651 7032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:31.109061 7032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:31.109143 7032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:31.109303 7032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:31.109375 7032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:31.144326 7032 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:33687
I20260929 02:15:31.144434 7159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:33687 every 8 connection(s)
I20260929 02:15:31.145872 7032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:15:31.155062 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 7032
I20260929 02:15:31.155290 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:15:31.155772 25442 external_mini_cluster.cc:1428] Setting key 7c7b700393a2521a1ca3411302593140
Sep 29 02:15:31 dist-test-slave-9k0h krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648131, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:31.168283 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.148909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:52759 (local address 127.24.216.190:43547)
0929 02:15:31.149383 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:31.149390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:31.150481 (+ 1091us) server_negotiation.cc:407] Connection header received
0929 02:15:31.152354 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:31.152358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:31.152431 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:31.152534 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:31.154098 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.155128 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.157019 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.157664 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.157809 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.161804 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:31.161810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:31.161814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:31.161851 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:31.164797 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:31.165717 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:31.165723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:31.165727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:31.165884 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:31.166396 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:31.166403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:31.166406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:31.166685 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:31.166860 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:31.167618 (+ 758us) server_negotiation.cc:299] Negotiation successful
0929 02:15:31.167876 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":224,"threads_started":1}
I20260929 02:15:31.169380 7163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.149525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:52759)
0929 02:15:31.150325 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:31.150371 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:31.152060 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:31.152798 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:31.152806 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:31.153332 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:31.153907 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:31.153925 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.155343 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.155351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:31.156885 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:31.156898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.157928 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.157974 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:31.158291 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.159021 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:31.159045 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:31.161350 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:31.164969 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:31.164979 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:31.165000 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:31.165582 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:31.166005 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:31.166011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:31.166014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:31.166285 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:31.166823 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:31.166835 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:31.167231 (+ 396us) client_negotiation.cc:769] Sending connection context
0929 02:15:31.167515 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 02:15:31.168137 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":269,"threads_started":1}
I20260929 02:15:31.171114 7160 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43547
I20260929 02:15:31.171459 7160 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:31.172338 7160 heartbeater.cc:511] Master 127.24.216.190:43547 requested a full tablet report, sending...
I20260929 02:15:31.174062 6837 ts_manager.cc:194] Registered new tserver with Master: a14fe4898e104f168d39c7f865cc170e (127.24.216.130:33687)
I20260929 02:15:31.175405 6837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:52759
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:15:31.214129 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43547
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:31.555313 7166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:31.555605 7166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:31.555756 7166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:31.563351 7166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:31.563472 7166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:31.563649 7166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:15:31.573796 7166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:31.575111 7166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:31.576411 7166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:31.590106 7171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:31.590076 7174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:31.590197 7172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:31.590842 7166 server_base.cc:1034] running on GCE node
I20260929 02:15:31.591537 7166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:31.592491 7166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:31.593719 7166 hybrid_clock.cc:648] HybridClock initialized: now 1790648131593669 us; error 46 us; skew 500 ppm
I20260929 02:15:31.594038 7166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:31 dist-test-slave-9k0h krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648131, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:31.597617 7166 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:15:31.599313 7166 webserver.cc:460] Webserver started at http://127.24.216.131:46303/ using document root <none> and password file <none>
I20260929 02:15:31.600076 7166 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:31.600220 7166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:31.600544 7166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:15:31.603600 7166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "dd62bbe3703f4c938cb98d1a01e708df"
format_stamp: "Formatted at 2026-09-29 02:15:31 on dist-test-slave-9k0h"
server_key: "57118a76488d7f5b4232f363f1471af3"
server_key_iv: "794eb5641974b2bc54e5409f1819852a"
server_key_version: "encryptionkey@0"
I20260929 02:15:31.604408 7166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "dd62bbe3703f4c938cb98d1a01e708df"
format_stamp: "Formatted at 2026-09-29 02:15:31 on dist-test-slave-9k0h"
server_key: "57118a76488d7f5b4232f363f1471af3"
server_key_iv: "794eb5641974b2bc54e5409f1819852a"
server_key_version: "encryptionkey@0"
I20260929 02:15:31.609773 7166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:15:31.613282 7181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:31.614682 7166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:15:31.614853 7166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "dd62bbe3703f4c938cb98d1a01e708df"
format_stamp: "Formatted at 2026-09-29 02:15:31 on dist-test-slave-9k0h"
server_key: "57118a76488d7f5b4232f363f1471af3"
server_key_iv: "794eb5641974b2bc54e5409f1819852a"
server_key_version: "encryptionkey@0"
I20260929 02:15:31.615060 7166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:31.628664 7166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:31.630052 7026 heartbeater.cc:503] Master 127.24.216.190:43547 was elected leader, sending a full tablet report...
I20260929 02:15:31.633244 7166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:31.633481 7166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:31.634403 7166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:31.635659 7166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:31.635728 7166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:31.635809 7166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:31.635855 7166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:31.667662 7166 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:36339
I20260929 02:15:31.667708 7293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:36339 every 8 connection(s)
I20260929 02:15:31.668994 7166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:15:31.671355 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 7166
I20260929 02:15:31.671581 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:15:31.672137 25442 external_mini_cluster.cc:1428] Setting key 7d3ba05c62a755716818d949db6d30d9
Sep 29 02:15:31 dist-test-slave-9k0h krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648131, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:31.690438 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.671557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:53467 (local address 127.24.216.190:43547)
0929 02:15:31.672116 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:31.672120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:31.673326 (+ 1206us) server_negotiation.cc:407] Connection header received
0929 02:15:31.675642 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:31.675646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:31.675708 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:31.675792 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:31.677206 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.678097 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.679818 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.680295 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.680399 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.684032 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:31.684037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:31.684040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:31.684088 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:31.686995 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:31.687943 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:31.687950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:31.687954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:31.688144 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:31.688637 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:31.688642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:31.688645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:31.688923 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:31.689084 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:31.689790 (+ 706us) server_negotiation.cc:299] Negotiation successful
0929 02:15:31.690026 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":273,"threads_started":1}
I20260929 02:15:31.691427 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.672258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:53467)
0929 02:15:31.673148 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:31.673213 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:31.675389 (+ 2176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:31.675987 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:31.675995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:31.676481 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:31.677037 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:31.677054 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.678295 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.678299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:31.679674 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:31.679686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.680543 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.680548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:31.680753 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.681517 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:31.681540 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:31.683812 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:31.687176 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:31.687183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:31.687215 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:31.687823 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:31.688267 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:31.688272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:31.688276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:31.688516 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:31.689050 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:31.689058 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:31.689424 (+ 366us) client_negotiation.cc:769] Sending connection context
0929 02:15:31.689694 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 02:15:31.690296 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":338,"threads_started":1}
I20260929 02:15:31.692811 7294 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43547
I20260929 02:15:31.693130 7294 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:31.693907 7294 heartbeater.cc:511] Master 127.24.216.190:43547 requested a full tablet report, sending...
I20260929 02:15:31.695390 6837 ts_manager.cc:194] Registered new tserver with Master: dd62bbe3703f4c938cb98d1a01e708df (127.24.216.131:36339)
I20260929 02:15:31.696204 6837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:53467
I20260929 02:15:31.700732 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:15:31.718228 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.705735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.24.216.190:43547)
0929 02:15:31.705991 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:31.705998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:31.706406 (+ 408us) server_negotiation.cc:407] Connection header received
0929 02:15:31.706543 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:31.706547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:31.706632 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:31.706723 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:31.708128 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.708859 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.710245 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.710672 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.710779 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.713111 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:31.713116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:31.713119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:31.713152 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:31.715647 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:31.716352 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:31.716356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:31.716359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:31.716483 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:31.716858 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:31.716862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:31.716868 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:31.717223 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:31.717423 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:31.717664 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 02:15:31.717841 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 02:15:31.722348 6837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:15:31.729224 6837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:15:31.760468 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.741682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:36339 (local address 127.0.0.1:53436)
0929 02:15:31.743315 (+ 1633us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:31.743369 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:31.743528 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:31.746312 (+ 2784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:31.746323 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:31.746381 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:31.746939 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:31.746956 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.751886 (+ 4930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.751896 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:31.755215 (+ 3319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:31.755238 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.759227 (+ 3989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.759237 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:31.759532 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.759621 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 02:15:31.759762 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 02:15:31.759977 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1406,"thread_start_us":2497,"threads_started":1}
I20260929 02:15:31.761416 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.741921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.24.216.131:36339)
0929 02:15:31.744168 (+ 2247us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:31.744179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:31.744217 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:15:31.744326 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:31.744334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:31.744605 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:31.747262 (+ 2657us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:31.747610 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.751674 (+ 4064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.756797 (+ 5123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.758912 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.760461 (+ 1549us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.760686 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:31.760968 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 02:15:31.761175 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2041,"thread_start_us":1893,"threads_started":1}
I20260929 02:15:31.764539 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.741867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:42215 (local address 127.0.0.1:45960)
0929 02:15:31.744431 (+ 2564us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:31.744453 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:31.744570 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:31.756468 (+ 11898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:31.756477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:31.756526 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:31.757077 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:31.757091 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.760198 (+ 3107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.760203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:31.762361 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:31.762374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.763779 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.763785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:31.763948 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.763978 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:15:31.764089 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 02:15:31.764225 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2446,"thread_start_us":1070,"threads_started":1}
I20260929 02:15:31.764708 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.742533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.24.216.129:42215)
0929 02:15:31.755381 (+ 12848us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:31.755394 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:31.755435 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:15:31.755583 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:31.755596 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:31.755932 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:31.757326 (+ 1394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:31.757673 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.759792 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.762590 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.763571 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.763728 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.763840 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:31.764159 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 02:15:31.764341 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12597,"thread_start_us":13598,"threads_started":1}
I20260929 02:15:31.765645 7229 tablet_service.cc:1467] Processing CreateTablet for tablet fbb55afc9f3e48d0856d252a5c4afbb2 (DEFAULT_TABLE table=test-table [id=0181f06267014f41b39160f964aa0bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:15:31.767174 7229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb55afc9f3e48d0856d252a5c4afbb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:31.768611 6961 tablet_service.cc:1467] Processing CreateTablet for tablet fbb55afc9f3e48d0856d252a5c4afbb2 (DEFAULT_TABLE table=test-table [id=0181f06267014f41b39160f964aa0bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:15:31.769984 6961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb55afc9f3e48d0856d252a5c4afbb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:31.770043 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.744869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:33687 (local address 127.0.0.1:40192)
0929 02:15:31.760737 (+ 15868us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:31.760760 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:31.760854 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:31.762381 (+ 1527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:31.762385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:31.762411 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:31.762590 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:31.762601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.765601 (+ 3000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.765606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:31.767486 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:31.767501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.769312 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.769318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:31.769484 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.769519 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:15:31.769599 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 02:15:31.769742 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15769,"thread_start_us":15237,"threads_started":1}
I20260929 02:15:31.771117 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.744883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.24.216.130:33687)
0929 02:15:31.760111 (+ 15228us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:31.760123 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:31.761850 (+ 1727us) server_negotiation.cc:407] Connection header received
0929 02:15:31.761939 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:31.761945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:31.762191 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:31.762395 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:31.763931 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.765455 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.767681 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.769116 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.770380 (+ 1264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.770547 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:31.770616 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 02:15:31.770785 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15054,"thread_start_us":15251,"threads_started":1}
I20260929 02:15:31.774705 7095 tablet_service.cc:1467] Processing CreateTablet for tablet fbb55afc9f3e48d0856d252a5c4afbb2 (DEFAULT_TABLE table=test-table [id=0181f06267014f41b39160f964aa0bb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:15:31.776168 7095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb55afc9f3e48d0856d252a5c4afbb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:31.779937 7313 tablet_bootstrap.cc:492] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df: Bootstrap starting.
I20260929 02:15:31.783444 7313 tablet_bootstrap.cc:654] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:31.784940 7313 log.cc:824] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:31.785364 7314 tablet_bootstrap.cc:492] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0: Bootstrap starting.
I20260929 02:15:31.788710 7313 tablet_bootstrap.cc:492] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df: No bootstrap required, opened a new log
I20260929 02:15:31.788993 7313 ts_tablet_manager.cc:1397] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 02:15:31.789314 7314 tablet_bootstrap.cc:654] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:31.790407 7314 log.cc:824] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:31.791082 7316 tablet_bootstrap.cc:492] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e: Bootstrap starting.
I20260929 02:15:31.795601 7316 tablet_bootstrap.cc:654] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:31.796902 7316 log.cc:824] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:31.797741 7314 tablet_bootstrap.cc:492] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0: No bootstrap required, opened a new log
I20260929 02:15:31.798048 7314 ts_tablet_manager.cc:1397] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20260929 02:15:31.798048 7313 raft_consensus.cc:348] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } }
I20260929 02:15:31.798594 7313 raft_consensus.cc:374] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:31.798722 7313 raft_consensus.cc:729] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd62bbe3703f4c938cb98d1a01e708df, State: Initialized, Role: FOLLOWER
I20260929 02:15:31.799319 7313 consensus_queue.cc:260] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } }
I20260929 02:15:31.800136 7316 tablet_bootstrap.cc:492] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e: No bootstrap required, opened a new log
I20260929 02:15:31.800406 7316 ts_tablet_manager.cc:1397] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 02:15:31.800508 7294 heartbeater.cc:503] Master 127.24.216.190:43547 was elected leader, sending a full tablet report...
I20260929 02:15:31.802539 7313 ts_tablet_manager.cc:1428] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 02:15:31.808933 7314 raft_consensus.cc:348] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } }
I20260929 02:15:31.809428 7314 raft_consensus.cc:374] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:31.809526 7314 raft_consensus.cc:729] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f865c9a4a7954672995d27c9079234b0, State: Initialized, Role: FOLLOWER
I20260929 02:15:31.810225 7314 consensus_queue.cc:260] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } }
I20260929 02:15:31.811090 7316 raft_consensus.cc:348] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } }
I20260929 02:15:31.811532 7316 raft_consensus.cc:374] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:31.811640 7316 raft_consensus.cc:729] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14fe4898e104f168d39c7f865cc170e, State: Initialized, Role: FOLLOWER
I20260929 02:15:31.812105 7314 ts_tablet_manager.cc:1428] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260929 02:15:31.812247 7316 consensus_queue.cc:260] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } }
I20260929 02:15:31.813280 7160 heartbeater.cc:503] Master 127.24.216.190:43547 was elected leader, sending a full tablet report...
I20260929 02:15:31.814080 7316 ts_tablet_manager.cc:1428] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260929 02:15:31.847704 7027 tablet.cc:2369] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:15:31.898129 7161 tablet.cc:2369] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:15:31.920740 7295 tablet.cc:2369] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:15:31.969286 7320 raft_consensus.cc:482] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:15:31.969641 7320 raft_consensus.cc:504] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } }
I20260929 02:15:31.971498 7320 leader_election.cc:288] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd62bbe3703f4c938cb98d1a01e708df (127.24.216.131:36339), a14fe4898e104f168d39c7f865cc170e (127.24.216.130:33687)
I20260929 02:15:31.978820 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.971989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:44209 (local address 127.24.216.131:36339)
0929 02:15:31.972217 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:31.972226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:31.972772 (+ 546us) server_negotiation.cc:407] Connection header received
0929 02:15:31.973124 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:31.973130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:31.973213 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:31.973287 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:31.974122 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.975059 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.976969 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.977805 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.977968 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.978026 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:31.978421 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 02:15:31.978568 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:15:31.979079 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.973049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:57105 (local address 127.24.216.130:33687)
0929 02:15:31.973289 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:31.973295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:31.973533 (+ 238us) server_negotiation.cc:407] Connection header received
0929 02:15:31.973731 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:31.973738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:31.973818 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:31.973924 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:31.974609 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.975434 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.977182 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.978054 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.978281 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.978358 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:31.978682 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 02:15:31.978836 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 02:15:31.978883 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.971939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:36339 (local address 127.24.216.129:44209)
0929 02:15:31.972605 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:31.972634 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:31.972903 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:31.973482 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:31.973486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:31.973536 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:31.973943 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:31.973955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.975187 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.975192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:31.976801 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:31.976812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.978021 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.978027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:31.978245 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.978283 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 02:15:31.978382 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 02:15:31.978488 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":363,"threads_started":1}
I20260929 02:15:31.979274 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:31.972798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:33687 (local address 127.24.216.129:57105)
0929 02:15:31.973431 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:31.973446 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:31.973600 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:31.973960 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:31.973964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:31.974008 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:31.974440 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:31.974459 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.975568 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.975572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:31.977013 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:31.977027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:31.978332 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:31.978337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:31.978510 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:31.978543 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:15:31.978642 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 02:15:31.978757 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"spinlock_wait_cycles":17280,"thread_start_us":314,"threads_started":1}
I20260929 02:15:31.980351 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb55afc9f3e48d0856d252a5c4afbb2" candidate_uuid: "f865c9a4a7954672995d27c9079234b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14fe4898e104f168d39c7f865cc170e" is_pre_election: true
I20260929 02:15:31.980350 7249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb55afc9f3e48d0856d252a5c4afbb2" candidate_uuid: "f865c9a4a7954672995d27c9079234b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd62bbe3703f4c938cb98d1a01e708df" is_pre_election: true
I20260929 02:15:31.980939 7249 raft_consensus.cc:2454] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f865c9a4a7954672995d27c9079234b0 in term 0.
I20260929 02:15:31.980927 7115 raft_consensus.cc:2454] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f865c9a4a7954672995d27c9079234b0 in term 0.
I20260929 02:15:31.981629 6916 leader_election.cc:302] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a14fe4898e104f168d39c7f865cc170e, f865c9a4a7954672995d27c9079234b0; no voters:
I20260929 02:15:31.982029 7320 raft_consensus.cc:2781] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:15:31.982206 7320 raft_consensus.cc:482] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:15:31.982326 7320 raft_consensus.cc:3037] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:31.984619 7320 raft_consensus.cc:504] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } }
I20260929 02:15:31.985466 7320 leader_election.cc:288] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [CANDIDATE]: Term 1 election: Requested vote from peers dd62bbe3703f4c938cb98d1a01e708df (127.24.216.131:36339), a14fe4898e104f168d39c7f865cc170e (127.24.216.130:33687)
I20260929 02:15:31.986028 7249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb55afc9f3e48d0856d252a5c4afbb2" candidate_uuid: "f865c9a4a7954672995d27c9079234b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd62bbe3703f4c938cb98d1a01e708df"
I20260929 02:15:31.986202 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb55afc9f3e48d0856d252a5c4afbb2" candidate_uuid: "f865c9a4a7954672995d27c9079234b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14fe4898e104f168d39c7f865cc170e"
I20260929 02:15:31.986306 7249 raft_consensus.cc:3037] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:31.986474 7115 raft_consensus.cc:3037] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:31.989890 7115 raft_consensus.cc:2454] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f865c9a4a7954672995d27c9079234b0 in term 1.
I20260929 02:15:31.989894 7249 raft_consensus.cc:2454] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f865c9a4a7954672995d27c9079234b0 in term 1.
I20260929 02:15:31.990422 6916 leader_election.cc:302] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd62bbe3703f4c938cb98d1a01e708df, f865c9a4a7954672995d27c9079234b0; no voters:
I20260929 02:15:31.990820 7320 raft_consensus.cc:2781] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:15:31.991330 7320 raft_consensus.cc:686] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [term 1 LEADER]: Becoming Leader. State: Replica: f865c9a4a7954672995d27c9079234b0, State: Running, Role: LEADER
I20260929 02:15:31.991822 7320 consensus_queue.cc:237] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } }
I20260929 02:15:31.996755 6836 catalog_manager.cc:5495] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f865c9a4a7954672995d27c9079234b0 (127.24.216.129). New cstate: current_term: 1 leader_uuid: "f865c9a4a7954672995d27c9079234b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } health_report { overall_health: UNKNOWN } } }
I20260929 02:15:32.051064 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:32.045707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.24.216.129:42215)
0929 02:15:32.045900 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:32.045905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:32.045928 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:15:32.046174 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:32.046178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:32.046271 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:32.046394 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:15:32.046765 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:32.047445 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:32.048880 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:32.049276 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:32.049363 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:32.049921 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:15:32.050467 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:15:32.050577 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:32.050694 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 02:15:32.050760 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:15:32.059775 7115 raft_consensus.cc:1261] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e [term 1 FOLLOWER]: Refusing update from remote peer f865c9a4a7954672995d27c9079234b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:15:32.059775 7249 raft_consensus.cc:1261] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df [term 1 FOLLOWER]: Refusing update from remote peer f865c9a4a7954672995d27c9079234b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:15:32.060860 7320 consensus_queue.cc:1035] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:15:32.061192 7324 consensus_queue.cc:1035] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:15:32.077889 7328 mvcc.cc:205] Tried to move back new op lower bound from 7334494748905832448 to 7334494748642148352. Current Snapshot: MvccSnapshot[applied={T|T < 7334494748905832448}]
I20260929 02:15:32.089520 6837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:15:32.089856 6837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:15:32.094408 6837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Sending DeleteTablet for 3 replicas of tablet fbb55afc9f3e48d0856d252a5c4afbb2
I20260929 02:15:32.095846 7095 tablet_service.cc:1514] Processing DeleteTablet for tablet fbb55afc9f3e48d0856d252a5c4afbb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:15:32 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:40192
I20260929 02:15:32.095929 6961 tablet_service.cc:1514] Processing DeleteTablet for tablet fbb55afc9f3e48d0856d252a5c4afbb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:15:32 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:45960
I20260929 02:15:32.096062 7229 tablet_service.cc:1514] Processing DeleteTablet for tablet fbb55afc9f3e48d0856d252a5c4afbb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:15:32 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:53436
I20260929 02:15:32.097137 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 6898
I20260929 02:15:32.106850 7339 tablet_replica.cc:330] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df: stopping tablet replica
I20260929 02:15:32.107617 7339 raft_consensus.cc:2229] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:15:32.108287 7339 raft_consensus.cc:2258] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:15:32.111616 7339 ts_tablet_manager.cc:1905] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:15:32.122566 7339 ts_tablet_manager.cc:1918] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:15:32.122972 7339 log.cc:1196] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/fbb55afc9f3e48d0856d252a5c4afbb2
I20260929 02:15:32.124039 7339 ts_tablet_manager.cc:1939] T fbb55afc9f3e48d0856d252a5c4afbb2 P dd62bbe3703f4c938cb98d1a01e708df: Deleting consensus metadata
I20260929 02:15:32.127141 6823 catalog_manager.cc:4841] TS dd62bbe3703f4c938cb98d1a01e708df (127.24.216.131:36339): tablet fbb55afc9f3e48d0856d252a5c4afbb2 (table test-table [id=0181f06267014f41b39160f964aa0bb1]) successfully deleted
I20260929 02:15:32.129914 7338 tablet_replica.cc:330] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0: stopping tablet replica
I20260929 02:15:32.130564 7338 raft_consensus.cc:2229] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:15:32.131278 7338 raft_consensus.cc:2258] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:15:32.134220 7338 ts_tablet_manager.cc:1905] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:15:32.140132 7337 tablet_replica.cc:330] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e: stopping tablet replica
I20260929 02:15:32.140650 7337 raft_consensus.cc:2229] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:15:32.141183 7337 raft_consensus.cc:2258] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:15:32.142256 7338 ts_tablet_manager.cc:1918] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:15:32.142442 7338 log.cc:1196] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/fbb55afc9f3e48d0856d252a5c4afbb2
I20260929 02:15:32.143066 7338 ts_tablet_manager.cc:1939] T fbb55afc9f3e48d0856d252a5c4afbb2 P f865c9a4a7954672995d27c9079234b0: Deleting consensus metadata
Sep 29 02:15:32 dist-test-slave-9k0h krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:15:32.145004 6822 catalog_manager.cc:4841] TS f865c9a4a7954672995d27c9079234b0 (127.24.216.129:42215): tablet fbb55afc9f3e48d0856d252a5c4afbb2 (table test-table [id=0181f06267014f41b39160f964aa0bb1]) successfully deleted
I20260929 02:15:32.146538 7337 ts_tablet_manager.cc:1905] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:15:32.154713 7337 ts_tablet_manager.cc:1918] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:15:32.154835 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:32.098335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.24.216.129:42215)
0929 02:15:32.098517 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:32.098523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:32.122243 (+ 23720us) server_negotiation.cc:407] Connection header received
0929 02:15:32.122869 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:32.122875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:32.122993 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:32.123084 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:32.125278 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:32.126674 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:32.129130 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:32.130675 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:32.130864 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:32.146655 (+ 15791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:32.146670 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:32.146685 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:32.146745 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:32.151095 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:32.152162 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:32.152173 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:32.152177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:32.152343 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:32.152778 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:32.152784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:32.152787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:32.153247 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:32.153537 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:32.154207 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 02:15:32.154461 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:15:32.155370 7337 log.cc:1196] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/fbb55afc9f3e48d0856d252a5c4afbb2
I20260929 02:15:32.156435 7337 ts_tablet_manager.cc:1939] T fbb55afc9f3e48d0856d252a5c4afbb2 P a14fe4898e104f168d39c7f865cc170e: Deleting consensus metadata
I20260929 02:15:32.156468 7021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:32.159544 6823 catalog_manager.cc:4841] TS a14fe4898e104f168d39c7f865cc170e (127.24.216.130:33687): tablet fbb55afc9f3e48d0856d252a5c4afbb2 (table test-table [id=0181f06267014f41b39160f964aa0bb1]) successfully deleted
I20260929 02:15:32.279826 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 6898
I20260929 02:15:32.298049 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 7032
Sep 29 02:15:32 dist-test-slave-9k0h krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:15:32.314826 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:32.298968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.24.216.130:33687)
0929 02:15:32.299189 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:32.299203 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:32.299225 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:15:32.299330 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:32.299335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:32.299406 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:32.299494 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:32.300828 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:32.301722 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:32.303547 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:32.304050 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:32.304175 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:32.308599 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:32.308616 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:32.308632 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:32.308692 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:32.311692 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:32.312360 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:32.312367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:32.312369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:32.312518 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:32.313055 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:32.313059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:32.313061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:32.313492 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:32.313721 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:32.314020 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 02:15:32.314382 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 02:15:32.315590 7155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:32.410411 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 7032
I20260929 02:15:32.429808 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 7166
Sep 29 02:15:32 dist-test-slave-9k0h krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:15:32.445922 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:32.430765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.24.216.131:36339)
0929 02:15:32.430941 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:32.430945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:32.431004 (+ 59us) server_negotiation.cc:407] Connection header received
0929 02:15:32.431261 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:32.431266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:32.431358 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:32.431467 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:32.432766 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:32.433553 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:32.434936 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:32.435354 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:32.435467 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:32.440070 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:32.440098 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:32.440115 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:32.440162 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:32.443252 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:32.443953 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:32.443961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:32.443963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:32.444134 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:32.444510 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:32.444514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:32.444516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:32.444896 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:32.445156 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:32.445478 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 02:15:32.445653 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:15:32.446617 7289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:32.537104 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 7166
I20260929 02:15:32.557011 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 6805
I20260929 02:15:32.558286 6867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:32.672317 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 6805
I20260929 02:15:32.694139 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:43547
--webserver_interface=127.24.216.190
--webserver_port=36887
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:43547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:33.034612 7345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:33.035178 7345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:33.035353 7345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:33.043001 7345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:15:33.043157 7345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:33.043244 7345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:33.043363 7345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:15:33.043439 7345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:15:33.053717 7345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:43547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:43547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=36887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:33.055100 7345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:33.056440 7345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:33.080267 7351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:33.080297 7354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:33.080268 7352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:33.080972 7345 server_base.cc:1034] running on GCE node
I20260929 02:15:33.081933 7345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:33.083557 7345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:33.084772 7345 hybrid_clock.cc:648] HybridClock initialized: now 1790648133084701 us; error 68 us; skew 500 ppm
I20260929 02:15:33.085189 7345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:33 dist-test-slave-9k0h krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648133, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:33.094521 7345 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:15:33.096354 7345 webserver.cc:460] Webserver started at http://127.24.216.190:36887/ using document root <none> and password file <none>
I20260929 02:15:33.097139 7345 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:33.097262 7345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:33.103628 7345 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 02:15:33.106891 7361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:33.108472 7345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:15:33.108670 7345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "f367efcbd04c450fa43cff72647fa490"
format_stamp: "Formatted at 2026-09-29 02:15:29 on dist-test-slave-9k0h"
server_key: "0aca06459ae82803e4f84e449ecea49c"
server_key_iv: "a47dc546e3fd908026c698649c357594"
server_key_version: "encryptionkey@0"
I20260929 02:15:33.109824 7345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:33.136775 7345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:33.141556 7345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:33.141856 7345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:33.160951 7345 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:43547
I20260929 02:15:33.160970 7412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:43547 every 8 connection(s)
I20260929 02:15:33.162586 7345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:15:33.169751 7413 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:15:33.170553 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 7345
I20260929 02:15:33.171360 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:42215
--local_ip_for_outbound_sockets=127.24.216.129
--tserver_master_addrs=127.24.216.190:43547
--webserver_port=39471
--webserver_interface=127.24.216.129
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:15:33.173319 7413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Bootstrap starting.
I20260929 02:15:33.198568 7413 log.cc:824] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:33.212255 7413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:15:33.212819 7413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Bootstrap complete.
I20260929 02:15:33.225314 7413 raft_consensus.cc:348] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } }
I20260929 02:15:33.226239 7413 raft_consensus.cc:729] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f367efcbd04c450fa43cff72647fa490, State: Initialized, Role: FOLLOWER
I20260929 02:15:33.226842 7413 consensus_queue.cc:260] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } }
I20260929 02:15:33.227105 7413 raft_consensus.cc:388] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:15:33.227234 7413 raft_consensus.cc:482] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:15:33.227415 7413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:15:33.231419 7413 raft_consensus.cc:504] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } }
I20260929 02:15:33.231969 7413 leader_election.cc:302] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f367efcbd04c450fa43cff72647fa490; no voters:
I20260929 02:15:33.232613 7413 leader_election.cc:288] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:15:33.232831 7417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:15:33.234172 7417 raft_consensus.cc:686] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 2 LEADER]: Becoming Leader. State: Replica: f367efcbd04c450fa43cff72647fa490, State: Running, Role: LEADER
I20260929 02:15:33.234958 7413 sys_catalog.cc:564] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:15:33.234859 7417 consensus_queue.cc:237] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } }
I20260929 02:15:33.239287 7418 sys_catalog.cc:455] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f367efcbd04c450fa43cff72647fa490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } } }
I20260929 02:15:33.239672 7418 sys_catalog.cc:458] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:33.240043 7419 sys_catalog.cc:455] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f367efcbd04c450fa43cff72647fa490. Latest consensus state: current_term: 2 leader_uuid: "f367efcbd04c450fa43cff72647fa490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } } }
I20260929 02:15:33.240325 7419 sys_catalog.cc:458] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:33.251557 7424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:15:33.261545 7424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:15:33.264057 7424 catalog_manager.cc:1259] Loaded cluster ID: 140f8011371446029a742427cdf3ceca
I20260929 02:15:33.264189 7424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:15:33.268345 7424 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 02:15:33.291500 7424 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 02:15:33.300827 7424 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:15:33.302623 7424 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Loaded TSK: 0
I20260929 02:15:33.304443 7424 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:15:33.538529 7415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:33.538813 7415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:33.538967 7415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:33.546262 7415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:33.546381 7415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:33.546584 7415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:15:33.556499 7415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:42215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=39471
--tserver_master_addrs=127.24.216.190:43547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:33.557833 7415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:33.559154 7415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:33.572763 7441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:33.573055 7442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:33.573026 7444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:33.574061 7415 server_base.cc:1034] running on GCE node
I20260929 02:15:33.574671 7415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:33.575676 7415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:33.576891 7415 hybrid_clock.cc:648] HybridClock initialized: now 1790648133576846 us; error 39 us; skew 500 ppm
I20260929 02:15:33.577198 7415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:33 dist-test-slave-9k0h krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648133, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:33.581002 7415 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:15:33.582628 7415 webserver.cc:460] Webserver started at http://127.24.216.129:39471/ using document root <none> and password file <none>
I20260929 02:15:33.583346 7415 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:33.583487 7415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:33.589048 7415 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:15:33.591897 7451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:33.593258 7415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:15:33.593432 7415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "f865c9a4a7954672995d27c9079234b0"
format_stamp: "Formatted at 2026-09-29 02:15:30 on dist-test-slave-9k0h"
server_key: "221133f7e657e03c4ce73c935caf98ed"
server_key_iv: "69d4708388696d2fa856fb18ca77e855"
server_key_version: "encryptionkey@0"
I20260929 02:15:33.594470 7415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:33.615463 7415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:33.619541 7415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:33.619884 7415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:33.620924 7415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:33.622424 7415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:33.622546 7415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:33.622673 7415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:33.622766 7415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:33.655470 7415 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:42215
I20260929 02:15:33.655553 7563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:42215 every 8 connection(s)
I20260929 02:15:33.656921 7415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:15:33.660838 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 7415
I20260929 02:15:33.661481 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:33687
--local_ip_for_outbound_sockets=127.24.216.130
--tserver_master_addrs=127.24.216.190:43547
--webserver_port=46683
--webserver_interface=127.24.216.130
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:15:33 dist-test-slave-9k0h krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648133, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:33.683015 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:33.659731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:47415 (local address 127.24.216.190:43547)
0929 02:15:33.660365 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:33.660379 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:33.661035 (+ 656us) server_negotiation.cc:407] Connection header received
0929 02:15:33.663156 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:33.663177 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:33.663629 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:33.664014 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:33.665371 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:33.667737 (+ 2366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:33.669651 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:33.670760 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:33.670915 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:33.675328 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:33.675337 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:33.675352 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:33.675394 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:33.678312 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:33.679074 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:33.679084 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:33.679096 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:33.679251 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:33.679691 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:33.679697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:33.679700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:33.680190 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:33.680391 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:33.681251 (+ 860us) server_negotiation.cc:299] Negotiation successful
0929 02:15:33.681634 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":279,"threads_started":1}
I20260929 02:15:33.683003 7567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:33.659964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:47415)
0929 02:15:33.660771 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:33.660820 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:33.662571 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:33.664133 (+ 1562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:33.664142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:33.664619 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:33.665145 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:33.665161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:33.667962 (+ 2801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:33.667967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:33.669447 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:33.669467 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:33.671027 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:33.671038 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:33.671377 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:33.672182 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:33.672222 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:33.675048 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:33.678477 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:33.678486 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:33.678501 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:33.678932 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:33.679373 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:33.679378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:33.679382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:33.679569 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:33.680336 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:33.680344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:33.680707 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 02:15:33.681046 (+ 339us) client_negotiation.cc:240] Negotiation successful
0929 02:15:33.681627 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":348,"threads_started":1}
I20260929 02:15:33.685858 7564 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43547
I20260929 02:15:33.686205 7564 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:33.686973 7564 heartbeater.cc:511] Master 127.24.216.190:43547 requested a full tablet report, sending...
I20260929 02:15:33.689069 7378 ts_manager.cc:194] Registered new tserver with Master: f865c9a4a7954672995d27c9079234b0 (127.24.216.129:42215)
I20260929 02:15:33.690649 7378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:47415
W20260929 02:15:34.000747 7568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:34.001075 7568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:34.001267 7568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:34.008596 7568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:34.008756 7568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:34.008925 7568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:15:34.019062 7568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:33687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=46683
--tserver_master_addrs=127.24.216.190:43547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:34.020479 7568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:34.021808 7568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:34.034880 7573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:34.034951 7576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:34.034921 7574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:34.035705 7568 server_base.cc:1034] running on GCE node
I20260929 02:15:34.036350 7568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:34.037303 7568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:34.038486 7568 hybrid_clock.cc:648] HybridClock initialized: now 1790648134038433 us; error 55 us; skew 500 ppm
I20260929 02:15:34.038800 7568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:34 dist-test-slave-9k0h krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648134, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:34.041929 7568 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:15:34.043507 7568 webserver.cc:460] Webserver started at http://127.24.216.130:46683/ using document root <none> and password file <none>
I20260929 02:15:34.044209 7568 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:34.044369 7568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:34.049594 7568 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:15:34.052435 7583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:34.053737 7568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:15:34.053910 7568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "a14fe4898e104f168d39c7f865cc170e"
format_stamp: "Formatted at 2026-09-29 02:15:31 on dist-test-slave-9k0h"
server_key: "56515a29b988783036896b3928731b6a"
server_key_iv: "375ffab8507042a2806a62f89587d5ac"
server_key_version: "encryptionkey@0"
I20260929 02:15:34.054947 7568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:34.070128 7568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:34.074203 7568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:34.074472 7568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:34.075438 7568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:34.076829 7568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:34.076947 7568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:34.077054 7568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:34.077142 7568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:34.112417 7568 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:33687
I20260929 02:15:34.112502 7695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:33687 every 8 connection(s)
I20260929 02:15:34.114049 7568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:15:34.118759 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 7568
I20260929 02:15:34.119557 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:36339
--local_ip_for_outbound_sockets=127.24.216.131
--tserver_master_addrs=127.24.216.190:43547
--webserver_port=46303
--webserver_interface=127.24.216.131
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:15:34 dist-test-slave-9k0h krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648134, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:34.138397 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:34.116777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:46215 (local address 127.24.216.190:43547)
0929 02:15:34.116968 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:34.116972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:34.118186 (+ 1214us) server_negotiation.cc:407] Connection header received
0929 02:15:34.121173 (+ 2987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:34.121177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:34.121256 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:34.121355 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:34.123583 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.124279 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.126801 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.127355 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.127490 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:34.131326 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:34.131335 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:34.131343 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:34.131389 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:34.134933 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:34.135677 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:34.135685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:34.135690 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:34.135859 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:34.136321 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:34.136329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:34.136334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:34.136774 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:34.136936 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:34.137610 (+ 674us) server_negotiation.cc:299] Negotiation successful
0929 02:15:34.137882 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:15:34.139015 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:34.117260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:46215)
0929 02:15:34.117975 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:34.118017 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:34.120836 (+ 2819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:34.121640 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:34.121655 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:34.122507 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:34.123373 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:34.123402 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.124440 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.124448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:34.126633 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:34.126650 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.127615 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.127624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:34.127949 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:34.128689 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:34.128712 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:34.131081 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:34.135129 (+ 4048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:34.135136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:34.135149 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:34.135549 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:34.135968 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:34.135971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:34.135973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:34.136215 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:34.136886 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:34.136892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:34.137213 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 02:15:34.137436 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 02:15:34.137829 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":286,"threads_started":1}
I20260929 02:15:34.140661 7696 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43547
I20260929 02:15:34.141033 7696 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:34.141821 7696 heartbeater.cc:511] Master 127.24.216.190:43547 requested a full tablet report, sending...
I20260929 02:15:34.143347 7378 ts_manager.cc:194] Registered new tserver with Master: a14fe4898e104f168d39c7f865cc170e (127.24.216.130:33687)
I20260929 02:15:34.144176 7378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:46215
W20260929 02:15:34.458775 7699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:34.459053 7699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:34.459244 7699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:34.467203 7699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:34.467377 7699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:34.467597 7699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:15:34.477515 7699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:36339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=46303
--tserver_master_addrs=127.24.216.190:43547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:34.478915 7699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:34.480274 7699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:34.494069 7707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:34.494103 7705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:34.494139 7704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:34.495481 7699 server_base.cc:1034] running on GCE node
I20260929 02:15:34.496120 7699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:34.497174 7699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:34.498390 7699 hybrid_clock.cc:648] HybridClock initialized: now 1790648134498332 us; error 47 us; skew 500 ppm
I20260929 02:15:34.498720 7699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:34 dist-test-slave-9k0h krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648134, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:34.501993 7699 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:15:34.503556 7699 webserver.cc:460] Webserver started at http://127.24.216.131:46303/ using document root <none> and password file <none>
I20260929 02:15:34.504292 7699 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:34.504446 7699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:34.509773 7699 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 02:15:34.512604 7714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:34.513931 7699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:15:34.514138 7699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "dd62bbe3703f4c938cb98d1a01e708df"
format_stamp: "Formatted at 2026-09-29 02:15:31 on dist-test-slave-9k0h"
server_key: "57118a76488d7f5b4232f363f1471af3"
server_key_iv: "794eb5641974b2bc54e5409f1819852a"
server_key_version: "encryptionkey@0"
I20260929 02:15:34.515159 7699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:34.551211 7699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:34.555745 7699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:34.556036 7699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:34.557042 7699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:34.558573 7699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:34.558686 7699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:34.558820 7699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:34.558903 7699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:34.592307 7699 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:36339
I20260929 02:15:34.592391 7826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:36339 every 8 connection(s)
I20260929 02:15:34.593806 7699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:15:34.599812 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 7699
Sep 29 02:15:34 dist-test-slave-9k0h krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648134, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:34.617906 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:34.600956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.24.216.190:43547)
0929 02:15:34.601467 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:34.601473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:34.602741 (+ 1268us) server_negotiation.cc:407] Connection header received
0929 02:15:34.602965 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:34.602971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:34.603043 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:34.603169 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:34.604977 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.605808 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.607192 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.607991 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.608141 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:34.611495 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:34.611500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:34.611503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:34.611560 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:34.614928 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:34.615829 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:34.615840 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:34.615843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:34.615968 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:34.616382 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:34.616387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:34.616389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:34.616702 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:34.616956 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:34.617242 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 02:15:34.617490 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":274,"threads_started":1}
I20260929 02:15:34.618001 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:34.596633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:37679 (local address 127.24.216.190:43547)
0929 02:15:34.596856 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:34.596861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:34.598138 (+ 1277us) server_negotiation.cc:407] Connection header received
0929 02:15:34.600609 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:34.600614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:34.600736 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:34.600843 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:34.602708 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.603602 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.605241 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.605690 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.605825 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:34.610029 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:34.610040 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:34.610046 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:34.610096 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:34.613920 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:34.615129 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:34.615138 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:34.615142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:34.615302 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:34.615774 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:34.615780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:34.615785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:34.616170 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:34.616329 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:34.617329 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0929 02:15:34.617653 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":94}
I20260929 02:15:34.619153 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:34.597149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:37679)
0929 02:15:34.597927 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:34.597963 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:34.600219 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:34.601129 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:34.601157 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:34.601815 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:34.602537 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:34.602563 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.603764 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.603770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:34.605103 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:34.605115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.605977 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.605986 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:34.606282 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:34.607041 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:34.607067 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:34.609776 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:34.614127 (+ 4351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:34.614138 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:34.614179 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:34.614896 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:34.615440 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:34.615443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:34.615445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:34.615648 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:34.616307 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:34.616331 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:34.616843 (+ 512us) client_negotiation.cc:769] Sending connection context
0929 02:15:34.617213 (+ 370us) client_negotiation.cc:240] Negotiation successful
0929 02:15:34.617779 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":338,"threads_started":1}
I20260929 02:15:34.620627 7827 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43547
I20260929 02:15:34.620895 7827 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:34.621593 7827 heartbeater.cc:511] Master 127.24.216.190:43547 requested a full tablet report, sending...
I20260929 02:15:34.622998 7378 ts_manager.cc:194] Registered new tserver with Master: dd62bbe3703f4c938cb98d1a01e708df (127.24.216.131:36339)
I20260929 02:15:34.623795 7378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:37679
I20260929 02:15:34.633067 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:15:34.640266 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:34.634071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.24.216.190:43547)
0929 02:15:34.634289 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:34.634293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:34.634767 (+ 474us) server_negotiation.cc:407] Connection header received
0929 02:15:34.634932 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:34.634935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:34.635070 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:34.635219 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:15:34.635723 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.636506 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.637953 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.638474 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.638585 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:34.639193 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:15:34.639737 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:15:34.639848 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:34.639963 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 02:15:34.640030 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 02:15:34.641239 7378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:15:34.648586 7378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:15:34.684345 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:34.661692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.24.216.129:42215)
0929 02:15:34.673877 (+ 12185us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:34.673886 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:34.674696 (+ 810us) server_negotiation.cc:407] Connection header received
0929 02:15:34.674805 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:34.674813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:34.675058 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:34.675263 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:34.676222 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.678183 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.681649 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.683129 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.683349 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:34.683529 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:34.683806 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 02:15:34.683970 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12018,"thread_start_us":11870,"threads_started":1}
I20260929 02:15:34.684446 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:34.661224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:42215 (local address 127.0.0.1:45980)
0929 02:15:34.674419 (+ 13195us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:34.674437 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:34.674578 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:34.675531 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:34.675536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:34.675564 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:34.676005 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:34.676017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.678345 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.678350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:34.681486 (+ 3136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:34.681501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.683340 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.683346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:34.683527 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:34.683587 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 02:15:34.683729 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 02:15:34.683921 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13079,"mutex_wait_us":35,"thread_start_us":12933,"threads_started":1}
I20260929 02:15:34.684569 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:34.660945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:33687 (local address 127.0.0.1:40222)
0929 02:15:34.674179 (+ 13234us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:34.674239 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:34.674461 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:34.675333 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:34.675341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:34.675373 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:34.675945 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:34.675969 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.678538 (+ 2569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.678547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:34.681510 (+ 2963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:34.681526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.683720 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.683726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:34.683880 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:34.683910 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:15:34.683989 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 02:15:34.684119 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12821,"mutex_wait_us":5,"thread_start_us":12545,"threads_started":1}
I20260929 02:15:34.685503 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:34.661088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.24.216.130:33687)
0929 02:15:34.673314 (+ 12226us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:34.673328 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:34.674677 (+ 1349us) server_negotiation.cc:407] Connection header received
0929 02:15:34.674786 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:34.674796 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:34.675071 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:34.675864 (+ 793us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:34.676243 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.678323 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.681707 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.683289 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.683460 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:34.683635 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:34.684959 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0929 02:15:34.685153 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11978,"thread_start_us":11788,"threads_started":1}
I20260929 02:15:34.688884 7499 tablet_service.cc:1467] Processing CreateTablet for tablet a8f1f1ed1c814d178ceb018458865ee5 (DEFAULT_TABLE table=test-table [id=03e9328506a444468ab6ed62ec60dd60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:15:34.688900 7631 tablet_service.cc:1467] Processing CreateTablet for tablet a8f1f1ed1c814d178ceb018458865ee5 (DEFAULT_TABLE table=test-table [id=03e9328506a444468ab6ed62ec60dd60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:15:34.689841 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:34.676196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:36339 (local address 127.0.0.1:53454)
0929 02:15:34.681510 (+ 5314us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:34.681529 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:34.681666 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:34.682427 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:34.682435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:34.682462 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:34.682861 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:34.682873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.685092 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.685104 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:34.687299 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:34.687311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.689030 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.689037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:34.689226 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:34.689260 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:15:34.689366 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 02:15:34.689502 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5157,"thread_start_us":4981,"threads_started":1}
I20260929 02:15:34.689901 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:34.675274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.24.216.131:36339)
0929 02:15:34.681134 (+ 5860us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:34.681143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:34.681788 (+ 645us) server_negotiation.cc:407] Connection header received
0929 02:15:34.681898 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:34.681906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:34.682222 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:34.683068 (+ 846us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:34.683355 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.684956 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.687457 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:34.688806 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:34.689012 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:34.689179 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:34.689410 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 02:15:34.689566 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5687,"thread_start_us":5520,"threads_started":1}
I20260929 02:15:34.690536 7631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f1f1ed1c814d178ceb018458865ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:34.690543 7499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f1f1ed1c814d178ceb018458865ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:34.693531 7564 heartbeater.cc:503] Master 127.24.216.190:43547 was elected leader, sending a full tablet report...
I20260929 02:15:34.693696 7762 tablet_service.cc:1467] Processing CreateTablet for tablet a8f1f1ed1c814d178ceb018458865ee5 (DEFAULT_TABLE table=test-table [id=03e9328506a444468ab6ed62ec60dd60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:15:34.695196 7762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f1f1ed1c814d178ceb018458865ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:34.706974 7843 tablet_bootstrap.cc:492] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0: Bootstrap starting.
I20260929 02:15:34.710652 7844 tablet_bootstrap.cc:492] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df: Bootstrap starting.
I20260929 02:15:34.710810 7843 tablet_bootstrap.cc:654] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:34.712095 7845 tablet_bootstrap.cc:492] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e: Bootstrap starting.
I20260929 02:15:34.712353 7843 log.cc:824] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:34.714947 7844 tablet_bootstrap.cc:654] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:34.716076 7845 tablet_bootstrap.cc:654] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:34.716248 7844 log.cc:824] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:34.717233 7843 tablet_bootstrap.cc:492] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0: No bootstrap required, opened a new log
I20260929 02:15:34.717243 7845 log.cc:824] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:34.717549 7843 ts_tablet_manager.cc:1397] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 02:15:34.719425 7844 tablet_bootstrap.cc:492] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df: No bootstrap required, opened a new log
I20260929 02:15:34.719712 7844 ts_tablet_manager.cc:1397] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 02:15:34.720264 7845 tablet_bootstrap.cc:492] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e: No bootstrap required, opened a new log
I20260929 02:15:34.720522 7845 ts_tablet_manager.cc:1397] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 02:15:34.725946 7843 raft_consensus.cc:348] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } }
I20260929 02:15:34.726392 7843 raft_consensus.cc:374] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:34.726496 7843 raft_consensus.cc:729] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f865c9a4a7954672995d27c9079234b0, State: Initialized, Role: FOLLOWER
I20260929 02:15:34.727111 7843 consensus_queue.cc:260] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } }
I20260929 02:15:34.729933 7843 ts_tablet_manager.cc:1428] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260929 02:15:34.730561 7844 raft_consensus.cc:348] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } }
I20260929 02:15:34.731019 7844 raft_consensus.cc:374] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:34.731137 7844 raft_consensus.cc:729] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd62bbe3703f4c938cb98d1a01e708df, State: Initialized, Role: FOLLOWER
I20260929 02:15:34.731069 7845 raft_consensus.cc:348] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } }
I20260929 02:15:34.731551 7845 raft_consensus.cc:374] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:34.731686 7845 raft_consensus.cc:729] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14fe4898e104f168d39c7f865cc170e, State: Initialized, Role: FOLLOWER
I20260929 02:15:34.731745 7844 consensus_queue.cc:260] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } }
I20260929 02:15:34.732291 7845 consensus_queue.cc:260] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } }
I20260929 02:15:34.733080 7827 heartbeater.cc:503] Master 127.24.216.190:43547 was elected leader, sending a full tablet report...
I20260929 02:15:34.733990 7844 ts_tablet_manager.cc:1428] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260929 02:15:34.739456 7696 heartbeater.cc:503] Master 127.24.216.190:43547 was elected leader, sending a full tablet report...
I20260929 02:15:34.740252 7845 ts_tablet_manager.cc:1428] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e: Time spent starting tablet: real 0.020s user 0.014s sys 0.006s
W20260929 02:15:34.845603 7828 tablet.cc:2369] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:15:34.866350 7697 tablet.cc:2369] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:15:34.909193 7565 tablet.cc:2369] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:15:35.019132 7849 raft_consensus.cc:482] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:15:35.019471 7849 raft_consensus.cc:504] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } }
I20260929 02:15:35.021246 7849 leader_election.cc:288] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a14fe4898e104f168d39c7f865cc170e (127.24.216.130:33687), dd62bbe3703f4c938cb98d1a01e708df (127.24.216.131:36339)
I20260929 02:15:35.027803 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:35.021753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:40995 (local address 127.24.216.130:33687)
0929 02:15:35.022059 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:35.022069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:35.022530 (+ 461us) server_negotiation.cc:407] Connection header received
0929 02:15:35.022775 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:35.022784 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:35.022886 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:35.023005 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:35.023623 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.024340 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.026069 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.026896 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.027071 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:35.027159 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:35.027417 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 02:15:35.027538 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:15:35.027956 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:35.021699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:33687 (local address 127.24.216.129:40995)
0929 02:15:35.022384 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:35.022400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:35.022585 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:35.023035 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:35.023041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:35.023076 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:35.023491 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:35.023500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.024468 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.024476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:35.025922 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:35.025934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.027094 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.027100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:35.027260 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:35.027283 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:15:35.027391 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 02:15:35.027517 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":1,"thread_start_us":348,"threads_started":1}
I20260929 02:15:35.028316 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:35.022726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:54367 (local address 127.24.216.131:36339)
0929 02:15:35.022939 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:35.022945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:35.023295 (+ 350us) server_negotiation.cc:407] Connection header received
0929 02:15:35.023457 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:35.023462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:35.023530 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:35.023647 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:35.024202 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.024866 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.026618 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.027405 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.027589 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:35.027662 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:35.027931 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 02:15:35.028069 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:15:35.028355 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:35.022527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:36339 (local address 127.24.216.129:54367)
0929 02:15:35.023160 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:35.023180 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:35.023313 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:35.023677 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:35.023682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:35.023709 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:35.024033 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:35.024042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.025008 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.025014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:35.026452 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:35.026469 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.027625 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.027631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:35.027777 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:35.027802 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 02:15:35.027908 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 02:15:35.028029 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":304,"threads_started":1}
I20260929 02:15:35.029213 7651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f1f1ed1c814d178ceb018458865ee5" candidate_uuid: "f865c9a4a7954672995d27c9079234b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14fe4898e104f168d39c7f865cc170e" is_pre_election: true
I20260929 02:15:35.029340 7782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f1f1ed1c814d178ceb018458865ee5" candidate_uuid: "f865c9a4a7954672995d27c9079234b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd62bbe3703f4c938cb98d1a01e708df" is_pre_election: true
I20260929 02:15:35.029728 7651 raft_consensus.cc:2454] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f865c9a4a7954672995d27c9079234b0 in term 0.
I20260929 02:15:35.029804 7782 raft_consensus.cc:2454] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f865c9a4a7954672995d27c9079234b0 in term 0.
I20260929 02:15:35.030417 7454 leader_election.cc:302] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a14fe4898e104f168d39c7f865cc170e, f865c9a4a7954672995d27c9079234b0; no voters:
I20260929 02:15:35.030782 7849 raft_consensus.cc:2781] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:15:35.030939 7849 raft_consensus.cc:482] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:15:35.031052 7849 raft_consensus.cc:3037] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:35.033196 7849 raft_consensus.cc:504] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } }
I20260929 02:15:35.033788 7849 leader_election.cc:288] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [CANDIDATE]: Term 1 election: Requested vote from peers a14fe4898e104f168d39c7f865cc170e (127.24.216.130:33687), dd62bbe3703f4c938cb98d1a01e708df (127.24.216.131:36339)
I20260929 02:15:35.034253 7651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f1f1ed1c814d178ceb018458865ee5" candidate_uuid: "f865c9a4a7954672995d27c9079234b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a14fe4898e104f168d39c7f865cc170e"
I20260929 02:15:35.034399 7782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f1f1ed1c814d178ceb018458865ee5" candidate_uuid: "f865c9a4a7954672995d27c9079234b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd62bbe3703f4c938cb98d1a01e708df"
I20260929 02:15:35.034447 7651 raft_consensus.cc:3037] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:35.034632 7782 raft_consensus.cc:3037] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:35.036824 7651 raft_consensus.cc:2454] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f865c9a4a7954672995d27c9079234b0 in term 1.
I20260929 02:15:35.037051 7782 raft_consensus.cc:2454] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f865c9a4a7954672995d27c9079234b0 in term 1.
I20260929 02:15:35.037262 7454 leader_election.cc:302] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a14fe4898e104f168d39c7f865cc170e, f865c9a4a7954672995d27c9079234b0; no voters:
I20260929 02:15:35.037549 7849 raft_consensus.cc:2781] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:15:35.037981 7849 raft_consensus.cc:686] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [term 1 LEADER]: Becoming Leader. State: Replica: f865c9a4a7954672995d27c9079234b0, State: Running, Role: LEADER
I20260929 02:15:35.038486 7849 consensus_queue.cc:237] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } } peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } }
I20260929 02:15:35.043253 7377 catalog_manager.cc:5495] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f865c9a4a7954672995d27c9079234b0 (127.24.216.129). New cstate: current_term: 1 leader_uuid: "f865c9a4a7954672995d27c9079234b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f865c9a4a7954672995d27c9079234b0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 } health_report { overall_health: UNKNOWN } } }
I20260929 02:15:35.141600 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:35.136165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45986 (local address 127.24.216.129:42215)
0929 02:15:35.136387 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:35.136393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:35.136413 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:15:35.136587 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:35.136591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:35.136685 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:35.136836 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:15:35.137209 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.137966 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.139405 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.139822 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.139955 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:35.140464 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:15:35.141009 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:15:35.141162 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:35.141264 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 02:15:35.141343 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:15:35.150210 7782 raft_consensus.cc:1261] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df [term 1 FOLLOWER]: Refusing update from remote peer f865c9a4a7954672995d27c9079234b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:15:35.150228 7651 raft_consensus.cc:1261] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e [term 1 FOLLOWER]: Refusing update from remote peer f865c9a4a7954672995d27c9079234b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:15:35.151151 7854 consensus_queue.cc:1035] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd62bbe3703f4c938cb98d1a01e708df" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:15:35.151526 7849 consensus_queue.cc:1035] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14fe4898e104f168d39c7f865cc170e" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:15:35.172166 7858 mvcc.cc:205] Tried to move back new op lower bound from 7334494761565351936 to 7334494761121492992. Current Snapshot: MvccSnapshot[applied={T|T < 7334494761565351936}]
I20260929 02:15:35.192219 7377 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:15:35.192524 7377 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:15:35.196245 7377 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Sending DeleteTablet for 3 replicas of tablet a8f1f1ed1c814d178ceb018458865ee5
I20260929 02:15:35.197531 7499 tablet_service.cc:1514] Processing DeleteTablet for tablet a8f1f1ed1c814d178ceb018458865ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:15:35 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:45980
I20260929 02:15:35.197588 7631 tablet_service.cc:1514] Processing DeleteTablet for tablet a8f1f1ed1c814d178ceb018458865ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:15:35 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:40222
I20260929 02:15:35.198367 7762 tablet_service.cc:1514] Processing DeleteTablet for tablet a8f1f1ed1c814d178ceb018458865ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:15:35 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:53454
I20260929 02:15:35.198669 7866 tablet_replica.cc:330] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0: stopping tablet replica
I20260929 02:15:35.198822 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 7415
I20260929 02:15:35.199160 7866 raft_consensus.cc:2229] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:15:35.199828 7866 raft_consensus.cc:2258] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:15:35.201468 7868 tablet_replica.cc:330] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df: stopping tablet replica
I20260929 02:15:35.202783 7866 ts_tablet_manager.cc:1905] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:15:35.203325 7868 raft_consensus.cc:2229] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:15:35.204427 7868 raft_consensus.cc:2258] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:15:35.210548 7868 ts_tablet_manager.cc:1905] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:15:35.210723 7867 tablet_replica.cc:330] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e: stopping tablet replica
I20260929 02:15:35.211172 7867 raft_consensus.cc:2229] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:15:35.211699 7867 raft_consensus.cc:2258] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:15:35.216307 7867 ts_tablet_manager.cc:1905] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:15:35.219215 7866 ts_tablet_manager.cc:1918] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:15:35.219381 7866 log.cc:1196] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/a8f1f1ed1c814d178ceb018458865ee5
I20260929 02:15:35.220050 7866 ts_tablet_manager.cc:1939] T a8f1f1ed1c814d178ceb018458865ee5 P f865c9a4a7954672995d27c9079234b0: Deleting consensus metadata
I20260929 02:15:35.222033 7363 catalog_manager.cc:4841] TS f865c9a4a7954672995d27c9079234b0 (127.24.216.129:42215): tablet a8f1f1ed1c814d178ceb018458865ee5 (table test-table [id=03e9328506a444468ab6ed62ec60dd60]) successfully deleted
I20260929 02:15:35.223816 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:35.199729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.24.216.129:42215)
0929 02:15:35.199904 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:35.199911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:35.200859 (+ 948us) server_negotiation.cc:407] Connection header received
0929 02:15:35.201075 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:35.201081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:35.201159 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:35.201261 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:35.204653 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.206047 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.210611 (+ 4564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.211343 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.211451 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:35.215543 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:35.215556 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:35.215574 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:35.215624 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:35.220042 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:35.221061 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:35.221070 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:35.221074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:35.221271 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:35.221762 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:35.221770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:35.221774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:35.222265 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:35.222569 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:35.223166 (+ 597us) server_negotiation.cc:299] Negotiation successful
0929 02:15:35.223427 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:15:35.225354 7559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:35.228600 7867 ts_tablet_manager.cc:1918] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:15:35.228775 7867 log.cc:1196] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/a8f1f1ed1c814d178ceb018458865ee5
I20260929 02:15:35.229306 7867 ts_tablet_manager.cc:1939] T a8f1f1ed1c814d178ceb018458865ee5 P a14fe4898e104f168d39c7f865cc170e: Deleting consensus metadata
I20260929 02:15:35.230991 7364 catalog_manager.cc:4841] TS a14fe4898e104f168d39c7f865cc170e (127.24.216.130:33687): tablet a8f1f1ed1c814d178ceb018458865ee5 (table test-table [id=03e9328506a444468ab6ed62ec60dd60]) successfully deleted
I20260929 02:15:35.231175 7868 ts_tablet_manager.cc:1918] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:15:35.231356 7868 log.cc:1196] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/a8f1f1ed1c814d178ceb018458865ee5
I20260929 02:15:35.231886 7868 ts_tablet_manager.cc:1939] T a8f1f1ed1c814d178ceb018458865ee5 P dd62bbe3703f4c938cb98d1a01e708df: Deleting consensus metadata
I20260929 02:15:35.233629 7364 catalog_manager.cc:4841] TS dd62bbe3703f4c938cb98d1a01e708df (127.24.216.131:36339): tablet a8f1f1ed1c814d178ceb018458865ee5 (table test-table [id=03e9328506a444468ab6ed62ec60dd60]) successfully deleted
I20260929 02:15:35.324056 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 7415
I20260929 02:15:35.342327 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 7568
I20260929 02:15:35.356108 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:35.343056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.24.216.130:33687)
0929 02:15:35.343250 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:35.343255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:35.343273 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:15:35.343417 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:35.343421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:35.343498 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:35.343588 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:35.345146 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.346128 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.347409 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.347796 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.347894 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:35.350524 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:35.350539 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:35.350555 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:35.350589 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:35.353277 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:35.354033 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:35.354041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:35.354043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:35.354269 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:35.354714 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:35.354718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:35.354720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:35.355106 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:35.355323 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:35.355576 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 02:15:35.355736 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:15:35.356741 7691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:35.448537 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 7568
I20260929 02:15:35.468144 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 7699
I20260929 02:15:35.482040 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:35.468936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.24.216.131:36339)
0929 02:15:35.469144 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:35.469151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:35.469245 (+ 94us) server_negotiation.cc:407] Connection header received
0929 02:15:35.469453 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:35.469457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:35.469533 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:35.469636 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:35.471085 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.471801 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.473140 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:35.473542 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:35.473669 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:35.476434 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:35.476450 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:35.476467 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:35.476521 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:35.479287 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:35.479992 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:35.479999 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:35.480002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:35.480143 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:35.480506 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:35.480519 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:35.480526 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:35.480871 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:35.481183 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:35.481485 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 02:15:35.481716 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:15:35.482743 7822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:35.573982 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 7699
I20260929 02:15:35.592110 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 7345
I20260929 02:15:35.593369 7408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:35.708388 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 7345
I20260929 02:15:35.730139 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:43547
--webserver_interface=127.24.216.190
--webserver_port=36887
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:43547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:36.062319 7874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:36.062671 7874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:36.062837 7874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:36.070012 7874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:15:36.070212 7874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:36.070338 7874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:36.070432 7874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:15:36.070515 7874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:15:36.080410 7874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:43547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:43547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=36887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:36.081804 7874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:36.083141 7874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:36.095016 7879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:36.095085 7880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:36.095057 7882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:36.096109 7874 server_base.cc:1034] running on GCE node
I20260929 02:15:36.097002 7874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:36.098647 7874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:36.099905 7874 hybrid_clock.cc:648] HybridClock initialized: now 1790648136099811 us; error 84 us; skew 500 ppm
I20260929 02:15:36.100443 7874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:36 dist-test-slave-9k0h krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648136, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:36.104236 7874 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:15:36.105922 7874 webserver.cc:460] Webserver started at http://127.24.216.190:36887/ using document root <none> and password file <none>
I20260929 02:15:36.106703 7874 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:36.106850 7874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:36.112469 7874 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 02:15:36.115579 7889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:36.117127 7874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:15:36.117331 7874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "f367efcbd04c450fa43cff72647fa490"
format_stamp: "Formatted at 2026-09-29 02:15:29 on dist-test-slave-9k0h"
server_key: "0aca06459ae82803e4f84e449ecea49c"
server_key_iv: "a47dc546e3fd908026c698649c357594"
server_key_version: "encryptionkey@0"
I20260929 02:15:36.118438 7874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:36.131237 7874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:36.135603 7874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:36.135907 7874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:36.154968 7874 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:43547
I20260929 02:15:36.155016 7940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:43547 every 8 connection(s)
I20260929 02:15:36.156678 7874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:15:36.165007 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 7874
I20260929 02:15:36.165892 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:42215
--local_ip_for_outbound_sockets=127.24.216.129
--tserver_master_addrs=127.24.216.190:43547
--webserver_port=39471
--webserver_interface=127.24.216.129
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:15:36.166079 7941 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:15:36.169865 7941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Bootstrap starting.
I20260929 02:15:36.189859 7941 log.cc:824] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:36.206874 7941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:15:36.207479 7941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490: Bootstrap complete.
I20260929 02:15:36.216751 7941 raft_consensus.cc:348] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } }
I20260929 02:15:36.217675 7941 raft_consensus.cc:729] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f367efcbd04c450fa43cff72647fa490, State: Initialized, Role: FOLLOWER
I20260929 02:15:36.218457 7941 consensus_queue.cc:260] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } }
I20260929 02:15:36.218700 7941 raft_consensus.cc:388] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:15:36.218819 7941 raft_consensus.cc:482] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:15:36.219004 7941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 2 FOLLOWER]: Advancing to term 3
I20260929 02:15:36.225232 7941 raft_consensus.cc:504] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } }
I20260929 02:15:36.225785 7941 leader_election.cc:302] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f367efcbd04c450fa43cff72647fa490; no voters:
I20260929 02:15:36.226470 7941 leader_election.cc:288] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 02:15:36.226624 7945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 02:15:36.227806 7945 raft_consensus.cc:686] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [term 3 LEADER]: Becoming Leader. State: Replica: f367efcbd04c450fa43cff72647fa490, State: Running, Role: LEADER
I20260929 02:15:36.228344 7945 consensus_queue.cc:237] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } }
I20260929 02:15:36.228765 7941 sys_catalog.cc:564] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:15:36.230907 7947 sys_catalog.cc:455] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f367efcbd04c450fa43cff72647fa490. Latest consensus state: current_term: 3 leader_uuid: "f367efcbd04c450fa43cff72647fa490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } } }
I20260929 02:15:36.231288 7947 sys_catalog.cc:458] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:36.231040 7946 sys_catalog.cc:455] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f367efcbd04c450fa43cff72647fa490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f367efcbd04c450fa43cff72647fa490" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43547 } } }
I20260929 02:15:36.231400 7946 sys_catalog.cc:458] T 00000000000000000000000000000000 P f367efcbd04c450fa43cff72647fa490 [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:36.243144 7952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:15:36.252396 7952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:15:36.254259 7952 catalog_manager.cc:1259] Loaded cluster ID: 140f8011371446029a742427cdf3ceca
I20260929 02:15:36.254427 7952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 02:15:36.256742 7952 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0f3e9059ad google::LogMessage::Fail() at ??:0
@ 0x7f0f3e90888a google::LogMessage::SendToLog() at ??:0
@ 0x7f0f3e9054de google::LogMessage::Flush() at ??:0
@ 0x7f0f3e906a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0f54338512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0f54335976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0f543bd610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0f543bd439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0f41551345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0f4155491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0f41554799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0f4152feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0f448096db start_thread at ??:0
@ 0x7f0f3d79f61f clone at ??:0
W20260929 02:15:36.568859 7943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:36.569160 7943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:36.569322 7943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:36.576905 7943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:36.577039 7943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:36.577224 7943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:15:36.587492 7943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:42215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=39471
--tserver_master_addrs=127.24.216.190:43547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:36.588927 7943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:36.590397 7943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:36.604498 7982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:36.604585 7983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:36.604604 7985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:36.605468 7943 server_base.cc:1034] running on GCE node
I20260929 02:15:36.606184 7943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:36.607188 7943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:36.608412 7943 hybrid_clock.cc:648] HybridClock initialized: now 1790648136608372 us; error 33 us; skew 500 ppm
I20260929 02:15:36.608744 7943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:36 dist-test-slave-9k0h krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648136, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:36.612087 7943 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:15:36.613667 7943 webserver.cc:460] Webserver started at http://127.24.216.129:39471/ using document root <none> and password file <none>
I20260929 02:15:36.614482 7943 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:36.614626 7943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:36.620282 7943 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 02:15:36.623459 7992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:36.624868 7943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:15:36.625079 7943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "f865c9a4a7954672995d27c9079234b0"
format_stamp: "Formatted at 2026-09-29 02:15:30 on dist-test-slave-9k0h"
server_key: "221133f7e657e03c4ce73c935caf98ed"
server_key_iv: "69d4708388696d2fa856fb18ca77e855"
server_key_version: "encryptionkey@0"
I20260929 02:15:36.626184 7943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:36.641004 7943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:36.645447 7943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:36.645735 7943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:36.646776 7943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:36.648093 7943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:36.648222 7943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:36.648351 7943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:36.648437 7943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:36.680675 7943 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:42215
I20260929 02:15:36.680733 8104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:42215 every 8 connection(s)
I20260929 02:15:36.682544 7943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:15:36.683216 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 7943
I20260929 02:15:36.684109 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:33687
--local_ip_for_outbound_sockets=127.24.216.130
--tserver_master_addrs=127.24.216.190:43547
--webserver_port=46683
--webserver_interface=127.24.216.130
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:15:36.688263 8107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:36.686371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:37949)
0929 02:15:36.687200 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:36.687418 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":328,"threads_started":1}
W20260929 02:15:36.693233 8105 heartbeater.cc:644] Failed to heartbeat to 127.24.216.190:43547 (0 consecutive failures): Network error: Failed to ping master at 127.24.216.190:43547: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
I20260929 02:15:36.694429 8107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:36.693866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:59235)
0929 02:15:36.694136 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:36.694291 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260929 02:15:36.695288 8107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:36.694945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:59581)
0929 02:15:36.695085 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:36.695169 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:15:36.695523 8105 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:15:37.038291 8108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:37.038604 8108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:37.038782 8108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:37.045877 8108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:37.046026 8108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:37.046221 8108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:15:37.055667 8108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:33687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=46683
--tserver_master_addrs=127.24.216.190:43547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:37.056903 8108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:37.058252 8108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:37.071168 8116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:37.071226 8113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:37.071250 8114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:37.075562 8108 server_base.cc:1034] running on GCE node
I20260929 02:15:37.076170 8108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:37.077138 8108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:37.078347 8108 hybrid_clock.cc:648] HybridClock initialized: now 1790648137078286 us; error 61 us; skew 500 ppm
I20260929 02:15:37.078647 8108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:37 dist-test-slave-9k0h krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648137, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:37.081669 8108 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:15:37.083230 8108 webserver.cc:460] Webserver started at http://127.24.216.130:46683/ using document root <none> and password file <none>
I20260929 02:15:37.083884 8108 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:37.083988 8108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:37.089268 8108 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 02:15:37.091948 8123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:37.093288 8108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:15:37.093461 8108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "a14fe4898e104f168d39c7f865cc170e"
format_stamp: "Formatted at 2026-09-29 02:15:31 on dist-test-slave-9k0h"
server_key: "56515a29b988783036896b3928731b6a"
server_key_iv: "375ffab8507042a2806a62f89587d5ac"
server_key_version: "encryptionkey@0"
I20260929 02:15:37.094452 8108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:37.112668 8108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:37.116978 8108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:37.117215 8108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:37.118254 8108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:37.119581 8108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:37.119683 8108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:37.119765 8108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:37.119822 8108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:37.150244 8108 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:33687
I20260929 02:15:37.150329 8235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:33687 every 8 connection(s)
I20260929 02:15:37.151773 8108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:15:37.156404 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:37.154664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:54071)
0929 02:15:37.155344 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:37.155558 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":269,"threads_started":1}
W20260929 02:15:37.157058 8236 heartbeater.cc:644] Failed to heartbeat to 127.24.216.190:43547 (0 consecutive failures): Network error: Failed to ping master at 127.24.216.190:43547: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
I20260929 02:15:37.157856 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:37.157541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:53133)
0929 02:15:37.157684 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:37.157762 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 02:15:37.158713 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:37.158399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:55291)
0929 02:15:37.158530 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:37.158609 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 02:15:37.158926 8236 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:15:37.161108 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 8108
I20260929 02:15:37.161762 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:36339
--local_ip_for_outbound_sockets=127.24.216.131
--tserver_master_addrs=127.24.216.190:43547
--webserver_port=46303
--webserver_interface=127.24.216.131
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:37.492359 8239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:37.492652 8239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:37.492820 8239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:37.500375 8239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:37.500500 8239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:37.500694 8239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:15:37.511409 8239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:36339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=46303
--tserver_master_addrs=127.24.216.190:43547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:37.512765 8239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:37.514140 8239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:37.528293 8244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:37.528326 8247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:37.528252 8245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:37.529640 8239 server_base.cc:1034] running on GCE node
I20260929 02:15:37.530393 8239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:37.531597 8239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:37.532836 8239 hybrid_clock.cc:648] HybridClock initialized: now 1790648137532780 us; error 48 us; skew 500 ppm
I20260929 02:15:37.533195 8239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:37 dist-test-slave-9k0h krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648137, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:37.536586 8239 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:15:37.538210 8239 webserver.cc:460] Webserver started at http://127.24.216.131:46303/ using document root <none> and password file <none>
I20260929 02:15:37.538908 8239 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:37.539049 8239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:37.544517 8239 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 02:15:37.547313 8254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:37.548623 8239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:15:37.548792 8239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "dd62bbe3703f4c938cb98d1a01e708df"
format_stamp: "Formatted at 2026-09-29 02:15:31 on dist-test-slave-9k0h"
server_key: "57118a76488d7f5b4232f363f1471af3"
server_key_iv: "794eb5641974b2bc54e5409f1819852a"
server_key_version: "encryptionkey@0"
I20260929 02:15:37.549775 8239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:37.563879 8239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:37.568187 8239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:37.568463 8239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:37.569473 8239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:37.571081 8239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:37.571225 8239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:37.571362 8239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:37.571451 8239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:37.606822 8239 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:36339
I20260929 02:15:37.606877 8366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:36339 every 8 connection(s)
I20260929 02:15:37.608295 8239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:15:37.613065 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:37.611311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:38047)
0929 02:15:37.612016 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:37.612240 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":270,"threads_started":1}
W20260929 02:15:37.613595 8367 heartbeater.cc:644] Failed to heartbeat to 127.24.216.190:43547 (0 consecutive failures): Network error: Failed to ping master at 127.24.216.190:43547: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
I20260929 02:15:37.614432 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:37.614051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:38231)
0929 02:15:37.614258 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:37.614328 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 02:15:37.615213 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:37.614914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:57423)
0929 02:15:37.615033 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:37.615109 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 02:15:37.615438 8367 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:15:37.616432 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 8239
I20260929 02:15:37.697111 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:37.696127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:60679)
0929 02:15:37.696633 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:37.696776 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":258,"threads_started":1}
I20260929 02:15:38.160359 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:38.159489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:54391)
0929 02:15:38.159979 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:38.160127 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":251,"threads_started":1}
I20260929 02:15:38.616958 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:38.616079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:40357)
0929 02:15:38.616583 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:38.616722 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":253,"threads_started":1}
I20260929 02:15:38.699054 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:38.698217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:49547)
0929 02:15:38.698704 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:38.698834 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":252,"threads_started":1}
I20260929 02:15:39.162032 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:39.161320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:52395)
0929 02:15:39.161795 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:39.161871 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":266,"threads_started":1}
I20260929 02:15:39.618800 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:39.618017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:60983)
0929 02:15:39.618546 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:39.618630 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":289,"threads_started":1}
I20260929 02:15:39.700641 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:39.699930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:43023)
0929 02:15:39.700384 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:39.700463 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":245,"threads_started":1}
I20260929 02:15:40.163599 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:40.162894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:48731)
0929 02:15:40.163337 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:40.163418 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
I20260929 02:15:40.620363 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:40.619611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:53801)
0929 02:15:40.620106 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:40.620188 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":285,"threads_started":1}
I20260929 02:15:40.702553 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:40.701621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:34177)
0929 02:15:40.702140 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:40.702318 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":234,"threads_started":1}
I20260929 02:15:41.165350 8381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:41.164551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:52283)
0929 02:15:41.165050 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:41.165149 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":279,"threads_started":1}
I20260929 02:15:41.622241 8382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:41.621378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:56093)
0929 02:15:41.621830 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:41.621927 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260929 02:15:41.704712 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:41.703779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:51767)
0929 02:15:41.704389 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:41.704477 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":355,"threads_started":1}
I20260929 02:15:42.167611 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:42.166602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:41515)
0929 02:15:42.167182 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:42.167357 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":293,"threads_started":1}
I20260929 02:15:42.623991 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:42.623194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:51009)
0929 02:15:42.623729 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:42.623821 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":296,"threads_started":1}
I20260929 02:15:42.706602 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:42.705711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:54757)
0929 02:15:42.706336 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:42.706431 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":262,"threads_started":1}
I20260929 02:15:43.169543 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:43.168748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:37363)
0929 02:15:43.169271 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:43.169374 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":282,"threads_started":1}
I20260929 02:15:43.626040 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:43.625087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:57771)
0929 02:15:43.625721 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:43.625860 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":368,"threads_started":1}
I20260929 02:15:43.708458 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:43.707628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:58477)
0929 02:15:43.708151 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:43.708242 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":237,"threads_started":1}
I20260929 02:15:44.171638 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:44.170685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:55105)
0929 02:15:44.171309 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:44.171400 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":317,"threads_started":1}
I20260929 02:15:44.628019 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:44.627156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:59497)
0929 02:15:44.627711 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:44.627801 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":240,"threads_started":1}
I20260929 02:15:44.710392 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:44.709516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:46951)
0929 02:15:44.710035 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:44.710132 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":263,"threads_started":1}
I20260929 02:15:45.173592 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:45.172830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:46819)
0929 02:15:45.173335 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:45.173427 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":264,"threads_started":1}
I20260929 02:15:45.630097 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:45.629158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:37509)
0929 02:15:45.629742 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:45.629824 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":323,"threads_started":1}
I20260929 02:15:45.712272 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:45.711416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:36161)
0929 02:15:45.711940 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:45.712027 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":260,"threads_started":1}
I20260929 02:15:46.175468 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:46.174709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:44479)
0929 02:15:46.175209 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:46.175293 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":260,"threads_started":1}
I20260929 02:15:46.632004 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:46.631155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:56717)
0929 02:15:46.631715 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:46.631819 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":283,"threads_started":1}
I20260929 02:15:46.713975 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:46.713257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:41523)
0929 02:15:46.713731 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:46.713816 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
I20260929 02:15:47.177263 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:47.176531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:57891)
0929 02:15:47.176987 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:47.177069 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":259,"threads_started":1}
I20260929 02:15:47.633618 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:47.632889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:53417)
0929 02:15:47.633371 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:47.633449 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":287,"threads_started":1}
I20260929 02:15:47.715715 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:47.714939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:36769)
0929 02:15:47.715438 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:47.715535 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":261,"threads_started":1}
I20260929 02:15:48.179299 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:48.178419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:32821)
0929 02:15:48.178954 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:48.179082 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":258,"threads_started":1}
I20260929 02:15:48.635506 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:48.634691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:40293)
0929 02:15:48.635217 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:48.635318 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":264,"threads_started":1}
I20260929 02:15:48.717590 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:48.716774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:51189)
0929 02:15:48.717328 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:48.717424 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":289,"threads_started":1}
I20260929 02:15:49.181092 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:49.180295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:35259)
0929 02:15:49.180805 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:49.180895 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":280,"threads_started":1}
I20260929 02:15:49.637238 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:49.636538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:36727)
0929 02:15:49.636989 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:49.637071 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":237,"threads_started":1}
I20260929 02:15:49.719573 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:49.718755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:58761)
0929 02:15:49.719287 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:49.719381 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":273,"threads_started":1}
I20260929 02:15:50.183081 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:50.182237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:58389)
0929 02:15:50.182790 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:50.182887 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":254,"threads_started":1}
I20260929 02:15:50.639004 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:50.638253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:34291)
0929 02:15:50.638727 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:50.638819 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":228,"threads_started":1}
I20260929 02:15:50.721632 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:50.720683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:45943)
0929 02:15:50.721300 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:50.721419 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":253,"threads_started":1}
I20260929 02:15:51.184909 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:51.184159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:42917)
0929 02:15:51.184638 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:51.184719 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":262,"threads_started":1}
I20260929 02:15:51.640730 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:51.639965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:45101)
0929 02:15:51.640443 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:51.640531 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":256,"threads_started":1}
I20260929 02:15:51.723863 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:51.722965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:57887)
0929 02:15:51.723558 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:51.723674 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":295,"threads_started":1}
I20260929 02:15:52.187143 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:52.186259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.130:43471)
0929 02:15:52.186833 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:52.186949 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":320,"threads_started":1}
I20260929 02:15:52.617794 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 7943
I20260929 02:15:52.637045 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:52.619138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.24.216.129:42215)
0929 02:15:52.619782 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:52.619812 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:52.619856 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:15:52.622133 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:52.622142 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:52.622652 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:52.622951 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:52.624490 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:52.625763 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:52.627165 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:52.627860 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:52.628003 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:52.631014 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:52.631024 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:52.631041 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:52.631080 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:52.634018 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:52.634708 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:52.634714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:52.634721 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:52.634886 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:52.635256 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:52.635259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:52.635261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:52.635785 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:52.636019 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:52.636406 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 02:15:52.636749 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":310,"threads_started":1}
I20260929 02:15:52.638644 8100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:52.642892 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:52.641995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.131:46381)
0929 02:15:52.642519 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:52.642641 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":278,"threads_started":1}
I20260929 02:15:52.733080 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:52.731775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43547 (local address 127.24.216.129:59463)
0929 02:15:52.732446 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:52.732739 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":316,"threads_started":1}
I20260929 02:15:52.737811 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 7943
I20260929 02:15:52.755833 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 8108
I20260929 02:15:52.774633 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:52.756811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40968 (local address 127.24.216.130:33687)
0929 02:15:52.757360 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:52.757375 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:52.757412 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:15:52.759158 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:52.759168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:52.759656 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:52.760052 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:52.761742 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:52.763075 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:52.764337 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:52.765007 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:52.765209 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:52.767888 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:52.767895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:52.767911 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:52.767949 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:52.771069 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:52.771817 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:52.771825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:52.771832 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:52.772007 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:52.772439 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:52.772444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:52.772447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:52.773002 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:52.773340 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:52.773824 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 02:15:52.774347 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":274,"threads_started":1}
I20260929 02:15:52.776139 8231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:52.863214 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 8108
I20260929 02:15:52.880931 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 8239
I20260929 02:15:52.900699 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:52.882259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.24.216.131:36339)
0929 02:15:52.882954 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:52.882969 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:52.883014 (+ 45us) server_negotiation.cc:407] Connection header received
0929 02:15:52.884689 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:52.884699 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:52.885153 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:52.885462 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:52.887074 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:52.888397 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:52.889810 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:52.890641 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:52.890796 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:52.893545 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:52.893554 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:52.893572 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:52.893628 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:52.896847 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:52.897627 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:52.897635 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:52.897649 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:52.897838 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:52.898538 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:52.898542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:52.898545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:52.899182 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:52.899448 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:52.900010 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 02:15:52.900390 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":344,"threads_started":1}
I20260929 02:15:52.902333 8362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:52.993309 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 8239
2026-09-29T02:15:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24930 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:15:53 dist-test-slave-9k0h krb5kdc[8425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:15:53 dist-test-slave-9k0h krb5kdc[8425](info): set up 2 sockets
Sep 29 02:15:53 dist-test-slave-9k0h krb5kdc[8425](info): commencing operation
krb5kdc: starting...
W20260929 02:15:54.122830 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:15:54 dist-test-slave-9k0h krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:15:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:15:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:15:54.389540 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:43751
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:43751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:54.716507 8438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:54.716863 8438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:54.717031 8438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:54.724200 8438 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:15:54.724334 8438 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:15:54.724406 8438 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:15:54.724498 8438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:15:54.724574 8438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:54.724702 8438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:54.724794 8438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:15:54.724875 8438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:15:54.734637 8438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:43751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:43751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:54.735987 8438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:54.737313 8438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:54.749033 8443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:54.749195 8444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:54.749063 8446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:54.750348 8438 server_base.cc:1034] running on GCE node
I20260929 02:15:54.751082 8438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:54.752386 8438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:54.753635 8438 hybrid_clock.cc:648] HybridClock initialized: now 1790648154753553 us; error 53 us; skew 500 ppm
I20260929 02:15:54.753968 8438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:54 dist-test-slave-9k0h krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648154, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:54.757244 8438 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:15:54.758720 8438 webserver.cc:460] Webserver started at http://127.24.216.190:41797/ using document root <none> and password file <none>
I20260929 02:15:54.759409 8438 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:54.759521 8438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:54.759860 8438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:15:54.762952 8438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "f411ba5e13c44711a03b3e6fdcd0eb68"
format_stamp: "Formatted at 2026-09-29 02:15:54 on dist-test-slave-9k0h"
server_key: "6cef6ca937bbad388cc40e0d20bf4bf1"
server_key_iv: "d8d07c97bfe785ad797bdae21bb31f8a"
server_key_version: "encryptionkey@0"
I20260929 02:15:54.763733 8438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "f411ba5e13c44711a03b3e6fdcd0eb68"
format_stamp: "Formatted at 2026-09-29 02:15:54 on dist-test-slave-9k0h"
server_key: "6cef6ca937bbad388cc40e0d20bf4bf1"
server_key_iv: "d8d07c97bfe785ad797bdae21bb31f8a"
server_key_version: "encryptionkey@0"
I20260929 02:15:54.768778 8438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:15:54.772212 8453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:54.773625 8438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:15:54.773816 8438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "f411ba5e13c44711a03b3e6fdcd0eb68"
format_stamp: "Formatted at 2026-09-29 02:15:54 on dist-test-slave-9k0h"
server_key: "6cef6ca937bbad388cc40e0d20bf4bf1"
server_key_iv: "d8d07c97bfe785ad797bdae21bb31f8a"
server_key_version: "encryptionkey@0"
I20260929 02:15:54.774047 8438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:54.787022 8438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:54.791020 8438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:54.791307 8438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:54.810232 8438 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:43751
I20260929 02:15:54.810286 8504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:43751 every 8 connection(s)
I20260929 02:15:54.811602 8438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:15:54.813995 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 8438
I20260929 02:15:54.814282 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:15:54.814975 25442 external_mini_cluster.cc:1428] Setting key 46c546831d918712a6ee24270a9561db
I20260929 02:15:54.815663 8505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:54.830492 8505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Bootstrap starting.
Sep 29 02:15:54 dist-test-slave-9k0h krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:54.834575 8505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:54.835952 8505 log.cc:824] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:54.838896 8505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: No bootstrap required, opened a new log
I20260929 02:15:54.840044 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:54.817163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.24.216.190:43751)
0929 02:15:54.817909 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:54.817920 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:54.817960 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:15:54.820282 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:54.820305 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:54.820686 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:54.820999 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:54.822874 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:54.824183 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:54.826115 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:54.826870 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:54.827003 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:54.831815 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:54.831826 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:54.831846 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:54.831891 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:54.835998 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:54.836752 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:54.836759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:54.836765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:54.836944 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:54.837344 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:54.837353 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:54.837358 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:54.837849 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:54.838210 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:54.838622 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 02:15:54.839061 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":359,"threads_started":1}
I20260929 02:15:54.847069 8505 raft_consensus.cc:348] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } }
I20260929 02:15:54.847460 8505 raft_consensus.cc:374] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:54.847579 8505 raft_consensus.cc:729] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f411ba5e13c44711a03b3e6fdcd0eb68, State: Initialized, Role: FOLLOWER
I20260929 02:15:54.848208 8505 consensus_queue.cc:260] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } }
I20260929 02:15:54.848461 8505 raft_consensus.cc:388] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:15:54.848620 8505 raft_consensus.cc:482] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:15:54.848793 8505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:54.851090 8505 raft_consensus.cc:504] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } }
I20260929 02:15:54.851604 8505 leader_election.cc:302] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f411ba5e13c44711a03b3e6fdcd0eb68; no voters:
I20260929 02:15:54.852262 8505 leader_election.cc:288] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:15:54.852422 8510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:15:54.853130 8510 raft_consensus.cc:686] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 1 LEADER]: Becoming Leader. State: Replica: f411ba5e13c44711a03b3e6fdcd0eb68, State: Running, Role: LEADER
I20260929 02:15:54.853668 8510 consensus_queue.cc:237] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } }
I20260929 02:15:54.854497 8505 sys_catalog.cc:564] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:15:54.856839 8512 sys_catalog.cc:455] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f411ba5e13c44711a03b3e6fdcd0eb68. Latest consensus state: current_term: 1 leader_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } } }
I20260929 02:15:54.856840 8511 sys_catalog.cc:455] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } } }
I20260929 02:15:54.857337 8512 sys_catalog.cc:458] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:54.857333 8511 sys_catalog.cc:458] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: This master's current role is: LEADER
I20260929 02:15:54.859210 8515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:15:54.864564 8515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:15:54.873620 8515 catalog_manager.cc:1347] Generated new cluster ID: b1c75e5dfa434f0080908bb1ae4d127d
I20260929 02:15:54.873737 8515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:15:54.896871 8515 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:15:54.898293 8515 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:15:54.906610 8515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Generated new TSK 0
I20260929 02:15:54.907490 8515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:15:54.978775 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43751
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:55.320465 8531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:55.320749 8531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:55.320843 8531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:55.328079 8531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:55.328187 8531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:55.328298 8531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:15:55.338002 8531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:55.339298 8531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:55.340659 8531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:55.352589 8536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:55.353268 8537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:55.354240 8539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:55.355106 8531 server_base.cc:1034] running on GCE node
I20260929 02:15:55.355770 8531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:55.356727 8531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:55.357933 8531 hybrid_clock.cc:648] HybridClock initialized: now 1790648155357878 us; error 47 us; skew 500 ppm
I20260929 02:15:55.358285 8531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:55 dist-test-slave-9k0h krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648155, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:55.361738 8531 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:15:55.363232 8531 webserver.cc:460] Webserver started at http://127.24.216.129:36359/ using document root <none> and password file <none>
I20260929 02:15:55.363929 8531 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:55.364068 8531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:55.364367 8531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:15:55.367367 8531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "60cfdee59b004958a18f9a2736e6f5b3"
format_stamp: "Formatted at 2026-09-29 02:15:55 on dist-test-slave-9k0h"
server_key: "01c381e5a6e9b02b3d19d7a6c4ddd2d3"
server_key_iv: "c5aff054dbdb2b34289dbf58d60c7ab4"
server_key_version: "encryptionkey@0"
I20260929 02:15:55.368112 8531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "60cfdee59b004958a18f9a2736e6f5b3"
format_stamp: "Formatted at 2026-09-29 02:15:55 on dist-test-slave-9k0h"
server_key: "01c381e5a6e9b02b3d19d7a6c4ddd2d3"
server_key_iv: "c5aff054dbdb2b34289dbf58d60c7ab4"
server_key_version: "encryptionkey@0"
I20260929 02:15:55.373147 8531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:15:55.376562 8546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:55.377900 8531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:15:55.378103 8531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "60cfdee59b004958a18f9a2736e6f5b3"
format_stamp: "Formatted at 2026-09-29 02:15:55 on dist-test-slave-9k0h"
server_key: "01c381e5a6e9b02b3d19d7a6c4ddd2d3"
server_key_iv: "c5aff054dbdb2b34289dbf58d60c7ab4"
server_key_version: "encryptionkey@0"
I20260929 02:15:55.378342 8531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:55.414929 8531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:55.419421 8531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:55.419745 8531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:55.420730 8531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:55.422287 8531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:55.422387 8531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:55.422549 8531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:55.422619 8531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:55.454995 8531 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:33141
I20260929 02:15:55.455089 8658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:33141 every 8 connection(s)
I20260929 02:15:55.456354 8531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:15:55.459511 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 8531
I20260929 02:15:55.459779 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:15:55.460726 25442 external_mini_cluster.cc:1428] Setting key 2be9abcf8cc39a011733fd8ceef7f8f9
Sep 29 02:15:55 dist-test-slave-9k0h krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648155, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:55.479308 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:55.458984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:60187 (local address 127.24.216.190:43751)
0929 02:15:55.459503 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:55.459509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:55.460606 (+ 1097us) server_negotiation.cc:407] Connection header received
0929 02:15:55.463005 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:55.463009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:55.463077 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:55.463166 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:55.465316 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:55.466025 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:55.468557 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:55.468960 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:55.469082 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:55.473428 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:55.473433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:55.473437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:55.473469 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:55.476102 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:55.476835 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:55.476843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:55.476851 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:55.476968 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:55.477424 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:55.477428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:55.477431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:55.477703 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:55.477911 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:55.478704 (+ 793us) server_negotiation.cc:299] Negotiation successful
0929 02:15:55.478937 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":252,"threads_started":1}
I20260929 02:15:55.480295 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:55.459527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:60187)
0929 02:15:55.460416 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:55.460475 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:55.462699 (+ 2224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:55.463452 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:55.463466 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:55.464269 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:55.465133 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:55.465159 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:55.466206 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:55.466215 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:55.468384 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:55.468403 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:55.469206 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:55.469215 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:55.469557 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:55.470426 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:55.470466 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:55.473190 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:55.476249 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:55.476258 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:55.476275 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:55.476700 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:55.477080 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:55.477085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:55.477089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:55.477299 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:55.477832 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:55.477840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:55.478271 (+ 431us) client_negotiation.cc:769] Sending connection context
0929 02:15:55.478574 (+ 303us) client_negotiation.cc:240] Negotiation successful
0929 02:15:55.479130 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":315,"threads_started":1}
I20260929 02:15:55.481920 8659 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43751
I20260929 02:15:55.482275 8659 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:55.483240 8659 heartbeater.cc:511] Master 127.24.216.190:43751 requested a full tablet report, sending...
I20260929 02:15:55.485649 8470 ts_manager.cc:194] Registered new tserver with Master: 60cfdee59b004958a18f9a2736e6f5b3 (127.24.216.129:33141)
I20260929 02:15:55.487190 8470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:60187
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:15:55.522536 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43751
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:55.867923 8665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:55.868247 8665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:55.868420 8665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:55.875716 8665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:55.875896 8665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:55.876076 8665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:15:55.886130 8665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:55.887508 8665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:55.888823 8665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:55.902021 8670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:55.902163 8671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:55.902082 8673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:55.902853 8665 server_base.cc:1034] running on GCE node
I20260929 02:15:55.903479 8665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:55.904431 8665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:55.905710 8665 hybrid_clock.cc:648] HybridClock initialized: now 1790648155905659 us; error 60 us; skew 500 ppm
I20260929 02:15:55.906086 8665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:55 dist-test-slave-9k0h krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648155, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:55.909271 8665 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:15:55.910851 8665 webserver.cc:460] Webserver started at http://127.24.216.130:34413/ using document root <none> and password file <none>
I20260929 02:15:55.911553 8665 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:55.911702 8665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:55.912010 8665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:15:55.915040 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "62b7e8e0afb24c948fdcc342df8b97c8"
format_stamp: "Formatted at 2026-09-29 02:15:55 on dist-test-slave-9k0h"
server_key: "e25ecf2d133e2bf74b1c3e78e3e27e1d"
server_key_iv: "4841d34682e617b453a3ee1b686fbefa"
server_key_version: "encryptionkey@0"
I20260929 02:15:55.915798 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "62b7e8e0afb24c948fdcc342df8b97c8"
format_stamp: "Formatted at 2026-09-29 02:15:55 on dist-test-slave-9k0h"
server_key: "e25ecf2d133e2bf74b1c3e78e3e27e1d"
server_key_iv: "4841d34682e617b453a3ee1b686fbefa"
server_key_version: "encryptionkey@0"
I20260929 02:15:55.920924 8665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:15:55.924495 8680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:55.925791 8665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:15:55.925966 8665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "62b7e8e0afb24c948fdcc342df8b97c8"
format_stamp: "Formatted at 2026-09-29 02:15:55 on dist-test-slave-9k0h"
server_key: "e25ecf2d133e2bf74b1c3e78e3e27e1d"
server_key_iv: "4841d34682e617b453a3ee1b686fbefa"
server_key_version: "encryptionkey@0"
I20260929 02:15:55.926242 8665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:55.955149 8665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:55.959869 8665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:55.960237 8665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:55.961284 8665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:55.962734 8665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:55.962839 8665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:55.962992 8665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:55.963084 8665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:55.998283 8665 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:44877
I20260929 02:15:55.998366 8792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:44877 every 8 connection(s)
I20260929 02:15:55.999599 8665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:15:56.008234 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 8665
I20260929 02:15:56.008448 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:15:56.009218 25442 external_mini_cluster.cc:1428] Setting key c874e507391401dd61361452c9c85437
Sep 29 02:15:56 dist-test-slave-9k0h krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648155, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:56.020399 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.002402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:44351 (local address 127.24.216.190:43751)
0929 02:15:56.002905 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:56.002911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:56.003740 (+ 829us) server_negotiation.cc:407] Connection header received
0929 02:15:56.005674 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:56.005680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:56.005781 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:56.005903 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:56.007512 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.008246 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.010250 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.010771 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.010912 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.014636 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:56.014641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:56.014644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:56.014676 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:56.017301 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:56.018037 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:56.018042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:56.018044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:56.018222 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:56.018649 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:56.018654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:56.018657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:56.018919 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:56.019085 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:56.019777 (+ 692us) server_negotiation.cc:299] Negotiation successful
0929 02:15:56.020003 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":390,"thread_start_us":285,"threads_started":1}
I20260929 02:15:56.021381 8796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.002883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:44351)
0929 02:15:56.003570 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:56.003615 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:56.005370 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:56.006195 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:56.006208 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:56.006766 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:56.007339 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:56.007356 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.008389 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.008394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:56.010073 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:56.010085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.011014 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.011022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:56.011328 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.012117 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:56.012141 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:56.014401 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:56.017448 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:56.017457 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:56.017474 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:56.017900 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:56.018328 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:56.018334 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:56.018338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:56.018525 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:56.019047 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:56.019057 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:56.019414 (+ 357us) client_negotiation.cc:769] Sending connection context
0929 02:15:56.019670 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 02:15:56.020235 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":282,"threads_started":1}
I20260929 02:15:56.022902 8793 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43751
I20260929 02:15:56.023216 8793 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:56.023924 8793 heartbeater.cc:511] Master 127.24.216.190:43751 requested a full tablet report, sending...
I20260929 02:15:56.025426 8470 ts_manager.cc:194] Registered new tserver with Master: 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877)
I20260929 02:15:56.026768 8470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:44351
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:15:56.067492 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43751
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:15:56.402083 8799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:15:56.402460 8799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:15:56.402647 8799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:15:56.410404 8799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:15:56.410521 8799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:15:56.410737 8799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:15:56.420738 8799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:43751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:15:56.422102 8799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:15:56.423556 8799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:15:56.436919 8805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:56.437048 8804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:15:56.436995 8807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:15:56.437971 8799 server_base.cc:1034] running on GCE node
I20260929 02:15:56.438603 8799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:15:56.439543 8799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:15:56.440744 8799 hybrid_clock.cc:648] HybridClock initialized: now 1790648156440708 us; error 42 us; skew 500 ppm
I20260929 02:15:56.441061 8799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:15:56 dist-test-slave-9k0h krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648156, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:15:56.444193 8799 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:15:56.445546 8799 webserver.cc:460] Webserver started at http://127.24.216.131:44227/ using document root <none> and password file <none>
I20260929 02:15:56.446244 8799 fs_manager.cc:359] Metadata directory not provided
I20260929 02:15:56.446348 8799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:15:56.446604 8799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:15:56.449365 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "174473aa2b4c4e269ac9dfdb2c95ec37"
format_stamp: "Formatted at 2026-09-29 02:15:56 on dist-test-slave-9k0h"
server_key: "b6b0f921f36db77ee2b9fbf796716138"
server_key_iv: "b0540a9892aed15027b2ccbe46b65a36"
server_key_version: "encryptionkey@0"
I20260929 02:15:56.450054 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "174473aa2b4c4e269ac9dfdb2c95ec37"
format_stamp: "Formatted at 2026-09-29 02:15:56 on dist-test-slave-9k0h"
server_key: "b6b0f921f36db77ee2b9fbf796716138"
server_key_iv: "b0540a9892aed15027b2ccbe46b65a36"
server_key_version: "encryptionkey@0"
I20260929 02:15:56.455224 8799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:15:56.458851 8814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:56.460364 8799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:15:56.460533 8799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "174473aa2b4c4e269ac9dfdb2c95ec37"
format_stamp: "Formatted at 2026-09-29 02:15:56 on dist-test-slave-9k0h"
server_key: "b6b0f921f36db77ee2b9fbf796716138"
server_key_iv: "b0540a9892aed15027b2ccbe46b65a36"
server_key_version: "encryptionkey@0"
I20260929 02:15:56.460772 8799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:15:56.483862 8799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:15:56.487839 8799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:15:56.488135 8799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:15:56.489104 8799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:15:56.490381 8659 heartbeater.cc:503] Master 127.24.216.190:43751 was elected leader, sending a full tablet report...
I20260929 02:15:56.490490 8799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:15:56.490623 8799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:56.490772 8799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:15:56.490847 8799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:15:56.523952 8799 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:46741
I20260929 02:15:56.523995 8926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:46741 every 8 connection(s)
I20260929 02:15:56.525246 8799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:15:56.533078 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 8799
I20260929 02:15:56.533383 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:15:56.533818 25442 external_mini_cluster.cc:1428] Setting key 9c9ad30bd9479d54c893d1ddbc5b4b12
Sep 29 02:15:56 dist-test-slave-9k0h krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648156, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:15:56.544970 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.527783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:46535 (local address 127.24.216.190:43751)
0929 02:15:56.528258 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:56.528263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:56.529057 (+ 794us) server_negotiation.cc:407] Connection header received
0929 02:15:56.531065 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:56.531071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:56.531173 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:56.531294 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:56.532786 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.533647 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.535652 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.536077 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.536209 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.539766 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:56.539771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:56.539774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:56.539806 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:56.542481 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:56.543108 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:56.543111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:56.543113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:56.543228 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:56.543575 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:56.543578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:56.543579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:56.543822 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:56.543934 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:56.544531 (+ 597us) server_negotiation.cc:299] Negotiation successful
0929 02:15:56.544690 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":246,"threads_started":1}
I20260929 02:15:56.545830 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.528187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:46535)
0929 02:15:56.528864 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:56.528904 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:56.530797 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:56.531523 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:56.531537 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:56.532047 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:15:56.532591 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:56.532608 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.533796 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.533801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:56.535500 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:56.535512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.536306 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.536313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:56.536569 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.537329 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:15:56.537351 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:15:56.539568 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:15:56.542596 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:56.542602 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:56.542613 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:56.543003 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:56.543313 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:15:56.543316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:15:56.543319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:15:56.543480 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:15:56.543919 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:15:56.543925 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:15:56.544230 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 02:15:56.544437 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 02:15:56.544790 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":266,"threads_started":1}
I20260929 02:15:56.547192 8927 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43751
I20260929 02:15:56.547544 8927 heartbeater.cc:464] Registering TS with master...
I20260929 02:15:56.548341 8927 heartbeater.cc:511] Master 127.24.216.190:43751 requested a full tablet report, sending...
I20260929 02:15:56.549752 8470 ts_manager.cc:194] Registered new tserver with Master: 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741)
I20260929 02:15:56.550601 8470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:46535
I20260929 02:15:56.561484 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:15:56.578994 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.566488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.24.216.190:43751)
0929 02:15:56.566727 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:56.566736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:56.567074 (+ 338us) server_negotiation.cc:407] Connection header received
0929 02:15:56.567223 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:56.567227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:56.567304 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:56.567384 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:56.568734 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.569480 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.570956 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.571361 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.571463 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.573777 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:56.573782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:56.573785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:56.573818 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:56.576430 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:56.577103 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:56.577109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:56.577113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:56.577245 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:56.577595 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:56.577598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:56.577601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:56.577971 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:56.578249 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:56.578498 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 02:15:56.578665 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:15:56.582959 8470 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:15:56.589661 8470 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:15:56.615916 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.602061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.24.216.130:44877)
0929 02:15:56.603354 (+ 1293us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:56.603365 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:56.604040 (+ 675us) server_negotiation.cc:407] Connection header received
0929 02:15:56.604468 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:56.604480 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:56.604778 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:56.604972 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:56.606233 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.608346 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.611974 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.613882 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.614059 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.614315 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:56.615214 (+ 899us) server_negotiation.cc:299] Negotiation successful
0929 02:15:56.615451 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1095,"thread_start_us":953,"threads_started":1}
I20260929 02:15:56.617301 8942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.601821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:44877 (local address 127.0.0.1:51678)
0929 02:15:56.603870 (+ 2049us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:56.603917 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:56.604211 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:56.605371 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:56.605382 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:56.605422 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:56.605956 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:56.605972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.608579 (+ 2607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.608587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:56.611740 (+ 3153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:56.611759 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.614556 (+ 2797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.614565 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:56.614863 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.614934 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 02:15:56.616534 (+ 1600us) client_negotiation.cc:240] Negotiation successful
0929 02:15:56.616803 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1738,"thread_start_us":2132,"threads_started":1}
I20260929 02:15:56.619985 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.602341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:46741 (local address 127.0.0.1:51094)
0929 02:15:56.604207 (+ 1866us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:56.604226 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:56.604428 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:56.606855 (+ 2427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:56.606863 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:56.606899 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:56.607385 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:56.607399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.611284 (+ 3885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.611293 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:56.615542 (+ 4249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:56.615570 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.619229 (+ 3659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.619235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:56.619355 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.619376 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 02:15:56.619494 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 02:15:56.619634 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1752,"thread_start_us":1058,"threads_started":1}
I20260929 02:15:56.620409 8943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.602909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.24.216.131:46741)
0929 02:15:56.603799 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:56.603808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:56.604315 (+ 507us) server_negotiation.cc:407] Connection header received
0929 02:15:56.604641 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:56.604649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:56.606238 (+ 1589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:56.606437 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:56.607634 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.611079 (+ 3445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.615773 (+ 4694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.617724 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.617937 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.619001 (+ 1064us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:56.619886 (+ 885us) server_negotiation.cc:299] Negotiation successful
0929 02:15:56.620057 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":1117,"threads_started":1}
I20260929 02:15:56.622380 8728 tablet_service.cc:1467] Processing CreateTablet for tablet 46bc0b28ac8e4c5798da05e3780a1a79 (DEFAULT_TABLE table=test-table [id=b57c5e14f12444489b83f0fd71289e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:15:56.623957 8728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46bc0b28ac8e4c5798da05e3780a1a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:56.623894 8862 tablet_service.cc:1467] Processing CreateTablet for tablet 46bc0b28ac8e4c5798da05e3780a1a79 (DEFAULT_TABLE table=test-table [id=b57c5e14f12444489b83f0fd71289e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:15:56.624893 8942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.604313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:33141 (local address 127.0.0.1:46874)
0929 02:15:56.617823 (+ 13510us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:56.617847 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:56.617966 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:56.619558 (+ 1592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:56.619562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:56.619589 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:56.619808 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:56.619822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.621600 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.621606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:56.622765 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:56.622776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.624160 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.624166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:56.624342 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.624401 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 02:15:56.624499 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 02:15:56.624621 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13376,"thread_start_us":13942,"threads_started":1}
I20260929 02:15:56.625507 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.604641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.24.216.129:33141)
0929 02:15:56.618919 (+ 14278us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:56.618927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:56.618954 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:15:56.619043 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:56.619050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:56.619315 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:56.619572 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:56.619979 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.621460 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.622912 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.623946 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.624161 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.624325 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:56.625034 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 02:15:56.625214 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14119,"thread_start_us":13950,"threads_started":1}
I20260929 02:15:56.625567 8862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46bc0b28ac8e4c5798da05e3780a1a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:56.628125 8594 tablet_service.cc:1467] Processing CreateTablet for tablet 46bc0b28ac8e4c5798da05e3780a1a79 (DEFAULT_TABLE table=test-table [id=b57c5e14f12444489b83f0fd71289e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:15:56.629249 8594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46bc0b28ac8e4c5798da05e3780a1a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:15:56.639328 8946 tablet_bootstrap.cc:492] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8: Bootstrap starting.
I20260929 02:15:56.640405 8947 tablet_bootstrap.cc:492] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37: Bootstrap starting.
I20260929 02:15:56.642381 8948 tablet_bootstrap.cc:492] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3: Bootstrap starting.
I20260929 02:15:56.644201 8947 tablet_bootstrap.cc:654] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:56.644662 8946 tablet_bootstrap.cc:654] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:56.645488 8947 log.cc:824] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:56.645581 8946 log.cc:824] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:56.645737 8948 tablet_bootstrap.cc:654] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3: Neither blocks nor log segments found. Creating new log.
I20260929 02:15:56.646955 8948 log.cc:824] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:15:56.648531 8947 tablet_bootstrap.cc:492] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37: No bootstrap required, opened a new log
I20260929 02:15:56.648656 8946 tablet_bootstrap.cc:492] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8: No bootstrap required, opened a new log
I20260929 02:15:56.648926 8946 ts_tablet_manager.cc:1397] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20260929 02:15:56.648944 8947 ts_tablet_manager.cc:1397] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260929 02:15:56.649816 8948 tablet_bootstrap.cc:492] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3: No bootstrap required, opened a new log
I20260929 02:15:56.650053 8948 ts_tablet_manager.cc:1397] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 02:15:56.657016 8947 raft_consensus.cc:348] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:15:56.657371 8947 raft_consensus.cc:374] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:56.657531 8947 raft_consensus.cc:729] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174473aa2b4c4e269ac9dfdb2c95ec37, State: Initialized, Role: FOLLOWER
I20260929 02:15:56.658188 8947 consensus_queue.cc:260] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:15:56.659291 8927 heartbeater.cc:503] Master 127.24.216.190:43751 was elected leader, sending a full tablet report...
I20260929 02:15:56.660039 8947 ts_tablet_manager.cc:1428] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260929 02:15:56.659886 8946 raft_consensus.cc:348] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:15:56.660315 8946 raft_consensus.cc:374] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:56.660482 8946 raft_consensus.cc:729] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62b7e8e0afb24c948fdcc342df8b97c8, State: Initialized, Role: FOLLOWER
I20260929 02:15:56.660473 8948 raft_consensus.cc:348] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:15:56.661026 8948 raft_consensus.cc:374] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:15:56.661190 8948 raft_consensus.cc:729] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60cfdee59b004958a18f9a2736e6f5b3, State: Initialized, Role: FOLLOWER
I20260929 02:15:56.661144 8946 consensus_queue.cc:260] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:15:56.661799 8948 consensus_queue.cc:260] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:15:56.662933 8946 ts_tablet_manager.cc:1428] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260929 02:15:56.663636 8948 ts_tablet_manager.cc:1428] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 02:15:56.664247 8793 heartbeater.cc:503] Master 127.24.216.190:43751 was elected leader, sending a full tablet report...
W20260929 02:15:56.708998 8660 tablet.cc:2369] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:15:56.733626 8953 raft_consensus.cc:482] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:15:56.733952 8953 raft_consensus.cc:504] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:15:56.735780 8953 leader_election.cc:288] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60cfdee59b004958a18f9a2736e6f5b3 (127.24.216.129:33141), 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741)
I20260929 02:15:56.742668 8943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.736458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:37809 (local address 127.24.216.131:46741)
0929 02:15:56.736710 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:56.736719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:56.737340 (+ 621us) server_negotiation.cc:407] Connection header received
0929 02:15:56.737540 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:56.737544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:56.737605 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:56.737704 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:56.738328 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.739053 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.740859 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.741611 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.741787 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.741873 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:56.742281 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 02:15:56.742410 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:15:56.742676 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.736235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:36273 (local address 127.24.216.129:33141)
0929 02:15:56.736516 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:56.736526 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:56.737203 (+ 677us) server_negotiation.cc:407] Connection header received
0929 02:15:56.737421 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:56.737424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:56.737488 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:56.737589 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:15:56.738241 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.738996 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.740865 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.741612 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.741809 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.741890 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:56.742264 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 02:15:56.742403 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:15:56.743040 8955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.736134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:46741 (local address 127.24.216.130:37809)
0929 02:15:56.737228 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:56.737242 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:56.737388 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:56.737719 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:56.737723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:56.737758 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:56.738124 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:56.738132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.739200 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.739207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:56.740705 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:56.740717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.741848 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.741855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:56.742019 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.742089 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 02:15:56.742241 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 02:15:56.742405 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":59,"thread_start_us":571,"threads_started":1}
I20260929 02:15:56.743114 8956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.736051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:33141 (local address 127.24.216.130:36273)
0929 02:15:56.737076 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0929 02:15:56.737091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:15:56.737268 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:15:56.737616 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:15:56.737620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:15:56.737644 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:15:56.738055 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:56.738065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.739125 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.739132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:56.740710 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:15:56.740721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.741886 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.741892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:15:56.742068 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.742090 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:15:56.742235 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 02:15:56.742414 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":63,"thread_start_us":885,"threads_started":1}
I20260929 02:15:56.744043 8882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46bc0b28ac8e4c5798da05e3780a1a79" candidate_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" is_pre_election: true
I20260929 02:15:56.744053 8614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46bc0b28ac8e4c5798da05e3780a1a79" candidate_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60cfdee59b004958a18f9a2736e6f5b3" is_pre_election: true
I20260929 02:15:56.744544 8614 raft_consensus.cc:2454] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62b7e8e0afb24c948fdcc342df8b97c8 in term 0.
I20260929 02:15:56.744539 8882 raft_consensus.cc:2454] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62b7e8e0afb24c948fdcc342df8b97c8 in term 0.
I20260929 02:15:56.745191 8683 leader_election.cc:302] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 174473aa2b4c4e269ac9dfdb2c95ec37, 62b7e8e0afb24c948fdcc342df8b97c8; no voters:
I20260929 02:15:56.745585 8953 raft_consensus.cc:2781] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:15:56.745724 8953 raft_consensus.cc:482] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:15:56.745831 8953 raft_consensus.cc:3037] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:56.748119 8953 raft_consensus.cc:504] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:15:56.748775 8953 leader_election.cc:288] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [CANDIDATE]: Term 1 election: Requested vote from peers 60cfdee59b004958a18f9a2736e6f5b3 (127.24.216.129:33141), 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741)
I20260929 02:15:56.749197 8614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46bc0b28ac8e4c5798da05e3780a1a79" candidate_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60cfdee59b004958a18f9a2736e6f5b3"
I20260929 02:15:56.749291 8882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46bc0b28ac8e4c5798da05e3780a1a79" candidate_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37"
I20260929 02:15:56.749413 8614 raft_consensus.cc:3037] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:15:56.749497 8882 raft_consensus.cc:3037] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 0 FOLLOWER]: Advancing to term 1
W20260929 02:15:56.751793 8794 tablet.cc:2369] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:15:56.752748 8882 raft_consensus.cc:2454] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62b7e8e0afb24c948fdcc342df8b97c8 in term 1.
I20260929 02:15:56.752751 8614 raft_consensus.cc:2454] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62b7e8e0afb24c948fdcc342df8b97c8 in term 1.
I20260929 02:15:56.753178 8684 leader_election.cc:302] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60cfdee59b004958a18f9a2736e6f5b3, 62b7e8e0afb24c948fdcc342df8b97c8; no voters:
I20260929 02:15:56.753485 8953 raft_consensus.cc:2781] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:15:56.753918 8953 raft_consensus.cc:686] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 1 LEADER]: Becoming Leader. State: Replica: 62b7e8e0afb24c948fdcc342df8b97c8, State: Running, Role: LEADER
I20260929 02:15:56.754495 8953 consensus_queue.cc:237] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:15:56.759400 8468 catalog_manager.cc:5495] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130). New cstate: current_term: 1 leader_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } health_report { overall_health: UNKNOWN } } }
W20260929 02:15:56.776918 8928 tablet.cc:2369] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:15:56.804822 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.799529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.24.216.130:44877)
0929 02:15:56.799787 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:56.799791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:56.799811 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:15:56.799976 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:56.799981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:56.800074 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:56.800198 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:15:56.800532 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.801263 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.802817 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.803213 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.803309 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.803732 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:15:56.804240 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:15:56.804355 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:56.804436 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 02:15:56.804504 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:15:56.813345 8882 raft_consensus.cc:1261] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 1 FOLLOWER]: Refusing update from remote peer 62b7e8e0afb24c948fdcc342df8b97c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:15:56.813344 8614 raft_consensus.cc:1261] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Refusing update from remote peer 62b7e8e0afb24c948fdcc342df8b97c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:15:56.814373 8957 consensus_queue.cc:1035] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:15:56.814743 8953 consensus_queue.cc:1035] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:15:56.826066 8960 mvcc.cc:205] Tried to move back new op lower bound from 7334494850296721408 to 7334494850071281664. Current Snapshot: MvccSnapshot[applied={T|T < 7334494850296721408}]
I20260929 02:15:56.827014 8961 mvcc.cc:205] Tried to move back new op lower bound from 7334494850296721408 to 7334494850071281664. Current Snapshot: MvccSnapshot[applied={T|T < 7334494850296721408}]
I20260929 02:15:56.849051 8468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:15:56.849462 8468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:15:56.854000 8468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Sending DeleteTablet for 3 replicas of tablet 46bc0b28ac8e4c5798da05e3780a1a79
I20260929 02:15:56.855315 8728 tablet_service.cc:1514] Processing DeleteTablet for tablet 46bc0b28ac8e4c5798da05e3780a1a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:15:56 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:51678
I20260929 02:15:56.855283 8594 tablet_service.cc:1514] Processing DeleteTablet for tablet 46bc0b28ac8e4c5798da05e3780a1a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:15:56 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:46874
I20260929 02:15:56.856150 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 8531
I20260929 02:15:56.856207 8971 tablet_replica.cc:330] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8: stopping tablet replica
I20260929 02:15:56.856611 8971 raft_consensus.cc:2229] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:15:56.856884 8862 tablet_service.cc:1514] Processing DeleteTablet for tablet 46bc0b28ac8e4c5798da05e3780a1a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:15:56 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:51094
I20260929 02:15:56.857275 8971 raft_consensus.cc:2258] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:15:56.857987 8970 tablet_replica.cc:330] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3: stopping tablet replica
I20260929 02:15:56.858440 8970 raft_consensus.cc:2229] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:15:56.858909 8970 raft_consensus.cc:2258] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:15:56.861056 8971 ts_tablet_manager.cc:1905] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:15:56.863348 8973 tablet_replica.cc:330] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37: stopping tablet replica
I20260929 02:15:56.863754 8973 raft_consensus.cc:2229] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:15:56.864126 8970 ts_tablet_manager.cc:1905] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:15:56.869442 8971 ts_tablet_manager.cc:1918] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:15:56.869624 8971 log.cc:1196] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/46bc0b28ac8e4c5798da05e3780a1a79
I20260929 02:15:56.870204 8971 ts_tablet_manager.cc:1939] T 46bc0b28ac8e4c5798da05e3780a1a79 P 62b7e8e0afb24c948fdcc342df8b97c8: Deleting consensus metadata
I20260929 02:15:56.870726 8973 raft_consensus.cc:2258] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:15:56.871853 8970 ts_tablet_manager.cc:1918] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:15:56.872054 8970 log.cc:1196] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/46bc0b28ac8e4c5798da05e3780a1a79
I20260929 02:15:56.872016 8457 catalog_manager.cc:4841] TS 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877): tablet 46bc0b28ac8e4c5798da05e3780a1a79 (table test-table [id=b57c5e14f12444489b83f0fd71289e29]) successfully deleted
I20260929 02:15:56.872591 8970 ts_tablet_manager.cc:1939] T 46bc0b28ac8e4c5798da05e3780a1a79 P 60cfdee59b004958a18f9a2736e6f5b3: Deleting consensus metadata
I20260929 02:15:56.873090 8973 ts_tablet_manager.cc:1905] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:15:56 dist-test-slave-9k0h krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:15:56.874260 8457 catalog_manager.cc:4841] TS 60cfdee59b004958a18f9a2736e6f5b3 (127.24.216.129:33141): tablet 46bc0b28ac8e4c5798da05e3780a1a79 (table test-table [id=b57c5e14f12444489b83f0fd71289e29]) successfully deleted
I20260929 02:15:56.878876 8973 ts_tablet_manager.cc:1918] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:15:56.879038 8973 log.cc:1196] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/46bc0b28ac8e4c5798da05e3780a1a79
I20260929 02:15:56.879519 8973 ts_tablet_manager.cc:1939] T 46bc0b28ac8e4c5798da05e3780a1a79 P 174473aa2b4c4e269ac9dfdb2c95ec37: Deleting consensus metadata
I20260929 02:15:56.881172 8456 catalog_manager.cc:4841] TS 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741): tablet 46bc0b28ac8e4c5798da05e3780a1a79 (table test-table [id=b57c5e14f12444489b83f0fd71289e29]) successfully deleted
I20260929 02:15:56.882947 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.857278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.24.216.129:33141)
0929 02:15:56.857476 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:56.857486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:56.863362 (+ 5876us) server_negotiation.cc:407] Connection header received
0929 02:15:56.863560 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:56.863615 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:56.863709 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:56.863816 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:56.865587 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.866452 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.867861 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:56.868302 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:56.868447 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:56.875038 (+ 6591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:56.875051 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:56.875069 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:56.875109 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:56.879078 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:56.879972 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:56.879981 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:56.879984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:56.880166 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:56.880601 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:56.880606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:56.880609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:56.881471 (+ 862us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:56.881779 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:56.882237 (+ 458us) server_negotiation.cc:299] Negotiation successful
0929 02:15:56.882526 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:15:56.883653 8654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:56.973412 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 8531
I20260929 02:15:56.996280 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 8665
Sep 29 02:15:57 dist-test-slave-9k0h krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:15:57.017019 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:56.997252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.24.216.130:44877)
0929 02:15:56.997452 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:56.997458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:56.997487 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:15:56.997589 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:56.997596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:56.997715 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:56.997839 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:56.999464 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:57.000724 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:57.002827 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:57.003585 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:57.003704 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:57.009412 (+ 5708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:57.009425 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:57.009444 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:57.009485 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:57.013610 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:57.014600 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:57.014609 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:57.014614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:57.014809 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:57.015233 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:57.015239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:57.015242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:57.015758 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:57.016113 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:57.016379 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 02:15:57.016618 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:15:57.017845 8788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:57.131174 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 8665
I20260929 02:15:57.151346 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 8799
Sep 29 02:15:57 dist-test-slave-9k0h krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:15:57.167461 8943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:15:57.152162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.24.216.131:46741)
0929 02:15:57.152349 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:15:57.152357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:15:57.152397 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:15:57.152613 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:15:57.152623 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:15:57.152696 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:15:57.152821 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:15:57.154081 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:57.154898 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:57.156311 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:15:57.156712 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:15:57.156851 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:15:57.161785 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:15:57.161803 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:15:57.161820 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:15:57.161854 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:15:57.164726 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:57.165504 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:57.165512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:57.165514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:57.165645 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:15:57.166039 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:15:57.166042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:15:57.166044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:15:57.166444 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:15:57.166673 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:15:57.166944 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 02:15:57.167106 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:15:57.168094 8922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:57.262599 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 8799
I20260929 02:15:57.280508 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 8438
I20260929 02:15:57.281708 8500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:15:57.390571 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 8438
I20260929 02:16:17.412416 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:43751
--webserver_interface=127.24.216.190
--webserver_port=41797
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:43751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:16:17.754328 8978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:16:17.754611 8978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:16:17.754700 8978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:16:17.761914 8978 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:16:17.762041 8978 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:16:17.762082 8978 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:16:17.762128 8978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:16:17.762238 8978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:16:17.762313 8978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:16:17.762358 8978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:16:17.762413 8978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:16:17.772573 8978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:43751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:43751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=41797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:16:17.773815 8978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:16:17.775177 8978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:16:17.797683 8984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:17.797869 8987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:17.797751 8985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:16:17.798763 8978 server_base.cc:1034] running on GCE node
I20260929 02:16:17.799654 8978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:16:17.801334 8978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:16:17.802596 8978 hybrid_clock.cc:648] HybridClock initialized: now 1790648177802531 us; error 53 us; skew 500 ppm
I20260929 02:16:17.803009 8978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:16:17 dist-test-slave-9k0h krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648177, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:16:17.812026 8978 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:16:17.813876 8978 webserver.cc:460] Webserver started at http://127.24.216.190:41797/ using document root <none> and password file <none>
I20260929 02:16:17.814800 8978 fs_manager.cc:359] Metadata directory not provided
I20260929 02:16:17.814911 8978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:16:17.821012 8978 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 02:16:17.824065 8994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:17.825534 8978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:16:17.825750 8978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "f411ba5e13c44711a03b3e6fdcd0eb68"
format_stamp: "Formatted at 2026-09-29 02:15:54 on dist-test-slave-9k0h"
server_key: "6cef6ca937bbad388cc40e0d20bf4bf1"
server_key_iv: "d8d07c97bfe785ad797bdae21bb31f8a"
server_key_version: "encryptionkey@0"
I20260929 02:16:17.826923 8978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:16:17.847242 8978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:16:17.851915 8978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:16:17.852237 8978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:16:17.872244 8978 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:43751
I20260929 02:16:17.872278 9045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:43751 every 8 connection(s)
I20260929 02:16:17.873878 8978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:16:17.879781 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 8978
I20260929 02:16:17.880687 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:33141
--local_ip_for_outbound_sockets=127.24.216.129
--tserver_master_addrs=127.24.216.190:43751
--webserver_port=36359
--webserver_interface=127.24.216.129
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:16:17.881418 9046 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:16:17.884899 9046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Bootstrap starting.
I20260929 02:16:17.905138 9046 log.cc:824] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Log is configured to *not* fsync() on all Append() calls
I20260929 02:16:17.917631 9046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:16:17.918263 9046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Bootstrap complete.
I20260929 02:16:17.927372 9046 raft_consensus.cc:348] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } }
I20260929 02:16:17.928241 9046 raft_consensus.cc:729] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f411ba5e13c44711a03b3e6fdcd0eb68, State: Initialized, Role: FOLLOWER
I20260929 02:16:17.928898 9046 consensus_queue.cc:260] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } }
I20260929 02:16:17.929155 9046 raft_consensus.cc:388] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:16:17.929307 9046 raft_consensus.cc:482] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:16:17.929500 9046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:16:17.933590 9046 raft_consensus.cc:504] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } }
I20260929 02:16:17.934135 9046 leader_election.cc:302] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f411ba5e13c44711a03b3e6fdcd0eb68; no voters:
I20260929 02:16:17.934823 9046 leader_election.cc:288] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:16:17.934957 9050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:16:17.936205 9050 raft_consensus.cc:686] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 2 LEADER]: Becoming Leader. State: Replica: f411ba5e13c44711a03b3e6fdcd0eb68, State: Running, Role: LEADER
I20260929 02:16:17.936822 9050 consensus_queue.cc:237] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } }
I20260929 02:16:17.937263 9046 sys_catalog.cc:564] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:16:17.939520 9052 sys_catalog.cc:455] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f411ba5e13c44711a03b3e6fdcd0eb68. Latest consensus state: current_term: 2 leader_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } } }
I20260929 02:16:17.939940 9052 sys_catalog.cc:458] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: This master's current role is: LEADER
I20260929 02:16:17.939495 9051 sys_catalog.cc:455] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } } }
I20260929 02:16:17.940686 9051 sys_catalog.cc:458] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: This master's current role is: LEADER
I20260929 02:16:17.944334 9056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:16:17.955874 9056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:16:17.957362 9056 catalog_manager.cc:1259] Loaded cluster ID: b1c75e5dfa434f0080908bb1ae4d127d
I20260929 02:16:17.957515 9056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:16:17.961827 9056 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:16:17.963641 9056 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Loaded TSK: 0
I20260929 02:16:18.010610 9056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Generated new TSK 1
I20260929 02:16:18.011082 9056 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:16:18.251489 9048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:16:18.251816 9048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:16:18.251984 9048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:16:18.260607 9048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:16:18.260766 9048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:16:18.260962 9048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:16:18.271646 9048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:33141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=36359
--tserver_master_addrs=127.24.216.190:43751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:16:18.273025 9048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:16:18.274449 9048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:16:18.289588 9073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:18.289784 9076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:18.290217 9074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:16:18.291091 9048 server_base.cc:1034] running on GCE node
I20260929 02:16:18.291764 9048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:16:18.292900 9048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:16:18.294260 9048 hybrid_clock.cc:648] HybridClock initialized: now 1790648178294185 us; error 57 us; skew 500 ppm
I20260929 02:16:18.294622 9048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:16:18 dist-test-slave-9k0h krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648178, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:16:18.297950 9048 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:16:18.299619 9048 webserver.cc:460] Webserver started at http://127.24.216.129:36359/ using document root <none> and password file <none>
I20260929 02:16:18.300326 9048 fs_manager.cc:359] Metadata directory not provided
I20260929 02:16:18.300473 9048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:16:18.306262 9048 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260929 02:16:18.309049 9083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:18.310616 9048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:16:18.310781 9048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "60cfdee59b004958a18f9a2736e6f5b3"
format_stamp: "Formatted at 2026-09-29 02:15:55 on dist-test-slave-9k0h"
server_key: "01c381e5a6e9b02b3d19d7a6c4ddd2d3"
server_key_iv: "c5aff054dbdb2b34289dbf58d60c7ab4"
server_key_version: "encryptionkey@0"
I20260929 02:16:18.311725 9048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:16:18.333489 9048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:16:18.338058 9048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:16:18.338402 9048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:16:18.339525 9048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:16:18.341022 9048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:16:18.341138 9048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:18.341276 9048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:16:18.341379 9048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:18.375505 9048 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:33141
I20260929 02:16:18.375554 9195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:33141 every 8 connection(s)
I20260929 02:16:18.376920 9048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:16:18.377840 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 9048
I20260929 02:16:18.378666 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:44877
--local_ip_for_outbound_sockets=127.24.216.130
--tserver_master_addrs=127.24.216.190:43751
--webserver_port=34413
--webserver_interface=127.24.216.130
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:16:18 dist-test-slave-9k0h krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648178, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:16:18.403916 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:18.380052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:38893 (local address 127.24.216.190:43751)
0929 02:16:18.380721 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:18.380739 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:18.381329 (+ 590us) server_negotiation.cc:407] Connection header received
0929 02:16:18.383613 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:18.383635 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:18.384089 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:18.384500 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:18.385972 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:18.387893 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:18.390002 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:18.390772 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:18.390915 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:18.395599 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:18.395608 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:18.395622 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:18.395662 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:18.398894 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:18.399969 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:18.399977 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:18.399983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:18.400178 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:18.400780 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:18.400785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:18.400787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:18.401324 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:18.401487 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:18.402589 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0929 02:16:18.402930 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":283,"threads_started":1}
I20260929 02:16:18.404474 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:18.380206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:38893)
0929 02:16:18.381137 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:18.381181 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:16:18.382989 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:16:18.384644 (+ 1655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:16:18.384652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:16:18.385207 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:16:18.385763 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:18.385781 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:18.388076 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:18.388089 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:18.389829 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:18.389847 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:18.391027 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:18.391037 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:18.391375 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:18.392151 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:16:18.392191 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:16:18.395237 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:16:18.399076 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:16:18.399088 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:16:18.399109 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:16:18.399793 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:16:18.400322 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:16:18.400330 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:16:18.400337 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:16:18.400627 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:16:18.401491 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:16:18.401503 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:16:18.402005 (+ 502us) client_negotiation.cc:769] Sending connection context
0929 02:16:18.402383 (+ 378us) client_negotiation.cc:240] Negotiation successful
0929 02:16:18.403045 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":406,"threads_started":1}
I20260929 02:16:18.407580 9196 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43751
I20260929 02:16:18.407922 9196 heartbeater.cc:464] Registering TS with master...
I20260929 02:16:18.408682 9196 heartbeater.cc:511] Master 127.24.216.190:43751 requested a full tablet report, sending...
I20260929 02:16:18.410923 9011 ts_manager.cc:194] Registered new tserver with Master: 60cfdee59b004958a18f9a2736e6f5b3 (127.24.216.129:33141)
I20260929 02:16:18.412380 9011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:38893
W20260929 02:16:18.730118 9200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:16:18.730533 9200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:16:18.730702 9200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:16:18.738751 9200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:16:18.738894 9200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:16:18.739085 9200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:16:18.750293 9200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:44877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=34413
--tserver_master_addrs=127.24.216.190:43751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:16:18.751753 9200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:16:18.753223 9200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:16:18.767534 9205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:18.767565 9206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:18.767892 9208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:16:18.768428 9200 server_base.cc:1034] running on GCE node
I20260929 02:16:18.769162 9200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:16:18.770321 9200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:16:18.771577 9200 hybrid_clock.cc:648] HybridClock initialized: now 1790648178771522 us; error 66 us; skew 500 ppm
I20260929 02:16:18.771951 9200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:16:18 dist-test-slave-9k0h krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648178, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:16:18.775452 9200 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:16:18.777024 9200 webserver.cc:460] Webserver started at http://127.24.216.130:34413/ using document root <none> and password file <none>
I20260929 02:16:18.777791 9200 fs_manager.cc:359] Metadata directory not provided
I20260929 02:16:18.777952 9200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:16:18.783506 9200 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:16:18.786396 9215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:18.787810 9200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:16:18.787995 9200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "62b7e8e0afb24c948fdcc342df8b97c8"
format_stamp: "Formatted at 2026-09-29 02:15:55 on dist-test-slave-9k0h"
server_key: "e25ecf2d133e2bf74b1c3e78e3e27e1d"
server_key_iv: "4841d34682e617b453a3ee1b686fbefa"
server_key_version: "encryptionkey@0"
I20260929 02:16:18.789000 9200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:16:18.803821 9200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:16:18.808409 9200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:16:18.808718 9200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:16:18.809782 9200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:16:18.811264 9200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:16:18.811389 9200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:18.811523 9200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:16:18.811614 9200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:18.846771 9200 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:44877
I20260929 02:16:18.846863 9327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:44877 every 8 connection(s)
I20260929 02:16:18.848496 9200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:16:18.848889 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 9200
I20260929 02:16:18.849637 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:46741
--local_ip_for_outbound_sockets=127.24.216.131
--tserver_master_addrs=127.24.216.190:43751
--webserver_port=44227
--webserver_interface=127.24.216.131
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:16:18 dist-test-slave-9k0h krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648178, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:16:18.873492 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:18.851456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:37197 (local address 127.24.216.190:43751)
0929 02:16:18.851702 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:18.851710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:18.852970 (+ 1260us) server_negotiation.cc:407] Connection header received
0929 02:16:18.854973 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:18.854982 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:18.855088 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:18.855219 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:18.857187 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:18.858355 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:18.860452 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:18.861262 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:18.861417 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:18.865776 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:18.865782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:18.865786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:18.865822 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:18.868695 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:18.869750 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:18.869755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:18.869758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:18.869893 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:18.870501 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:18.870505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:18.870508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:18.870832 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:18.871078 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:18.872643 (+ 1565us) server_negotiation.cc:299] Negotiation successful
0929 02:16:18.872892 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:16:18.875303 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:18.851934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:37197)
0929 02:16:18.852774 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:18.852828 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:16:18.854677 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:16:18.855414 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:16:18.855422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:16:18.855966 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:16:18.856959 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:18.856986 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:18.858520 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:18.858528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:18.860285 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:18.860302 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:18.861505 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:18.861511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:18.861727 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:18.862509 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:16:18.862533 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:16:18.865494 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:16:18.868866 (+ 3372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:16:18.868879 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:16:18.868901 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:16:18.869598 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:16:18.870036 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:16:18.870044 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:16:18.870051 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:16:18.870338 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:16:18.871001 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:16:18.871013 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:16:18.871786 (+ 773us) client_negotiation.cc:769] Sending connection context
0929 02:16:18.872504 (+ 718us) client_negotiation.cc:240] Negotiation successful
0929 02:16:18.873596 (+ 1092us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":361,"threads_started":1}
I20260929 02:16:18.883478 9328 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43751
I20260929 02:16:18.883839 9328 heartbeater.cc:464] Registering TS with master...
I20260929 02:16:18.884718 9328 heartbeater.cc:511] Master 127.24.216.190:43751 requested a full tablet report, sending...
I20260929 02:16:18.886509 9011 ts_manager.cc:194] Registered new tserver with Master: 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877)
I20260929 02:16:18.887784 9011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:37197
W20260929 02:16:19.205741 9331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:16:19.206041 9331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:16:19.206231 9331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:16:19.214005 9331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:16:19.214191 9331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:16:19.214423 9331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:16:19.228263 9331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:46741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=44227
--tserver_master_addrs=127.24.216.190:43751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:16:19.229791 9331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:16:19.231657 9331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:16:19.245682 9336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:19.245597 9339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:19.245651 9337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:16:19.246766 9331 server_base.cc:1034] running on GCE node
I20260929 02:16:19.247394 9331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:16:19.248432 9331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:16:19.249744 9331 hybrid_clock.cc:648] HybridClock initialized: now 1790648179249683 us; error 50 us; skew 500 ppm
I20260929 02:16:19.250109 9331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:16:19 dist-test-slave-9k0h krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648179, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:16:19.253424 9331 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:16:19.255055 9331 webserver.cc:460] Webserver started at http://127.24.216.131:44227/ using document root <none> and password file <none>
I20260929 02:16:19.255817 9331 fs_manager.cc:359] Metadata directory not provided
I20260929 02:16:19.255988 9331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:16:19.263197 9331 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:16:19.266467 9346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:19.267959 9331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:16:19.268168 9331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "174473aa2b4c4e269ac9dfdb2c95ec37"
format_stamp: "Formatted at 2026-09-29 02:15:56 on dist-test-slave-9k0h"
server_key: "b6b0f921f36db77ee2b9fbf796716138"
server_key_iv: "b0540a9892aed15027b2ccbe46b65a36"
server_key_version: "encryptionkey@0"
I20260929 02:16:19.269230 9331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:16:19.289381 9331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:16:19.294190 9331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:16:19.294486 9331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:16:19.295524 9331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:16:19.296926 9331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:16:19.297061 9331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:19.297197 9331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:16:19.297290 9331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:19.332661 9331 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:46741
I20260929 02:16:19.332751 9458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:46741 every 8 connection(s)
I20260929 02:16:19.334115 9331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:16:19.342031 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 9331
Sep 29 02:16:19 dist-test-slave-9k0h krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648179, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:16:19.356871 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.336979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:59851 (local address 127.24.216.190:43751)
0929 02:16:19.337210 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:19.337216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:19.338533 (+ 1317us) server_negotiation.cc:407] Connection header received
0929 02:16:19.340452 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:19.340456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:19.340524 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:19.340614 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:19.342492 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.343533 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.345237 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.345708 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.345871 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.350215 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:19.350224 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:19.350232 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:19.350283 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:19.353672 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:19.354482 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:19.354493 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:19.354496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:19.354628 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:19.355113 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:19.355116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:19.355118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:19.355397 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:19.355564 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:19.356270 (+ 706us) server_negotiation.cc:299] Negotiation successful
0929 02:16:19.356482 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 02:16:19.358237 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.337520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:59851)
0929 02:16:19.338380 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:19.338425 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:16:19.340179 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:16:19.340822 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:16:19.340830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:16:19.341382 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:16:19.342271 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:19.342298 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.343782 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.343790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:19.345090 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:19.345102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.345975 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.345984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:19.346296 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.347058 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:16:19.347098 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:16:19.349946 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:16:19.353826 (+ 3880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:16:19.353833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:16:19.353846 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:16:19.354338 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:16:19.354755 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:16:19.354767 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:16:19.354771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:16:19.354960 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:16:19.355534 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:16:19.355550 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:16:19.355903 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 02:16:19.356188 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 02:16:19.356739 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":346,"threads_started":1}
I20260929 02:16:19.358320 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.343167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.24.216.190:43751)
0929 02:16:19.343923 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:19.343930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:19.343945 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:16:19.343992 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:19.343996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:19.344073 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:19.344183 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:19.345861 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.346806 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.348851 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.349478 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.349585 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.352476 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:19.352483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:19.352488 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:19.352543 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:19.355524 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:19.356204 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:19.356212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:19.356215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:19.356335 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:19.356741 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:19.356747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:19.356750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:19.357239 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:19.357481 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:19.357799 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 02:16:19.357956 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":507,"threads_started":1}
I20260929 02:16:19.359956 9459 heartbeater.cc:347] Connected to a master server at 127.24.216.190:43751
I20260929 02:16:19.360296 9459 heartbeater.cc:464] Registering TS with master...
I20260929 02:16:19.361155 9459 heartbeater.cc:511] Master 127.24.216.190:43751 requested a full tablet report, sending...
I20260929 02:16:19.362571 9011 ts_manager.cc:194] Registered new tserver with Master: 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741)
I20260929 02:16:19.363451 9011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:59851
I20260929 02:16:19.373273 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:16:19.380963 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.374353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.24.216.190:43751)
0929 02:16:19.374607 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:19.374614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:19.374997 (+ 383us) server_negotiation.cc:407] Connection header received
0929 02:16:19.375250 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:19.375255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:19.375436 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:19.375604 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:16:19.375964 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.377067 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.378740 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.379423 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.379545 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.380068 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:16:19.380338 (+ 270us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 02:16:19.380699 (+ 361us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60688: token expired
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 02:16:19.381094 9463 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60688: token expired
W20260929 02:16:19.381583 8933 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.24.216.190:43751)
I20260929 02:16:19.395083 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.382034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.24.216.190:43751)
0929 02:16:19.382242 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:19.382246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:19.382260 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:16:19.382388 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:19.382391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:19.382453 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:19.382559 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:19.383907 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.384699 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.386076 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.386603 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.386713 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.389623 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:19.389630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:19.389633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:19.389674 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:19.392396 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:19.393093 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:19.393098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:19.393100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:19.393318 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:19.393672 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:19.393675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:19.393677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:19.394050 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:19.394296 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:19.394666 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 02:16:19.394845 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:16:19.398684 9011 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:16:19.406096 9011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:16:19.415927 9196 heartbeater.cc:503] Master 127.24.216.190:43751 was elected leader, sending a full tablet report...
I20260929 02:16:19.436630 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.418588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:44877 (local address 127.0.0.1:59830)
0929 02:16:19.419909 (+ 1321us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:19.419955 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:16:19.420140 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:16:19.427804 (+ 7664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:16:19.427812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:16:19.427840 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:16:19.428388 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:19.428400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.430783 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.430788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:19.433506 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:19.433525 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.435581 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.435587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:19.435818 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.435884 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 02:16:19.435987 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 02:16:19.436188 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"thread_start_us":8382,"threads_started":1}
I20260929 02:16:19.438234 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.418773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.24.216.130:44877)
0929 02:16:19.427104 (+ 8331us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:19.427113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:19.427147 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:16:19.427263 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:19.427274 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:19.427570 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:19.428625 (+ 1055us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:16:19.428857 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.430575 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.433777 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.435327 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.437441 (+ 2114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.437616 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:19.437688 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 02:16:19.437864 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8134,"thread_start_us":8001,"threads_started":1}
I20260929 02:16:19.438299 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.419795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.24.216.131:46741)
0929 02:16:19.426783 (+ 6988us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:19.426794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:19.427352 (+ 558us) server_negotiation.cc:407] Connection header received
0929 02:16:19.427648 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:19.427656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:19.427877 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:19.428047 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:16:19.428990 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.430594 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.434281 (+ 3687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.435750 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.435900 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.436085 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:19.437760 (+ 1675us) server_negotiation.cc:299] Negotiation successful
0929 02:16:19.437909 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6789,"thread_start_us":6631,"threads_started":1}
I20260929 02:16:19.439080 9469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.419105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:46741 (local address 127.0.0.1:60432)
0929 02:16:19.427238 (+ 8133us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:19.427257 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:16:19.427452 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:16:19.428397 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:16:19.428401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:16:19.428426 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:16:19.428795 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:19.428806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.431915 (+ 3109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.431920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:19.434099 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:19.434113 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.437388 (+ 3275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.437394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:19.437567 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.437596 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:16:19.438527 (+ 931us) client_negotiation.cc:240] Negotiation successful
0929 02:16:19.438696 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8017,"thread_start_us":980,"threads_started":1}
I20260929 02:16:19.443840 9394 tablet_service.cc:1467] Processing CreateTablet for tablet c243f3145650481fb33e67bfde272cb9 (DEFAULT_TABLE table=test-table [id=d5dcc955d6534806a9061ffe5d15a434]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:16:19.443840 9263 tablet_service.cc:1467] Processing CreateTablet for tablet c243f3145650481fb33e67bfde272cb9 (DEFAULT_TABLE table=test-table [id=d5dcc955d6534806a9061ffe5d15a434]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:16:19.445348 9263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c243f3145650481fb33e67bfde272cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:16:19.445348 9394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c243f3145650481fb33e67bfde272cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:16:19.446295 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.427393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:33141 (local address 127.0.0.1:48042)
0929 02:16:19.436865 (+ 9472us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:19.436884 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:16:19.436977 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:16:19.440970 (+ 3993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:16:19.440975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:16:19.441013 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:16:19.441197 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:19.441209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.442777 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.442781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:19.444123 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:19.444145 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.445522 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.445529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:19.445706 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.445748 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 02:16:19.445865 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 02:16:19.445998 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9378,"thread_start_us":12475,"threads_started":1}
I20260929 02:16:19.446424 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.427638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48042 (local address 127.24.216.129:33141)
0929 02:16:19.440389 (+ 12751us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:19.440397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:19.440427 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:16:19.440535 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:19.440542 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:19.440791 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:19.440989 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:16:19.441351 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.442630 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.444289 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.445337 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.445542 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.445710 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:19.445908 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 02:16:19.446074 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12576,"thread_start_us":12433,"threads_started":1}
I20260929 02:16:19.449425 9131 tablet_service.cc:1467] Processing CreateTablet for tablet c243f3145650481fb33e67bfde272cb9 (DEFAULT_TABLE table=test-table [id=d5dcc955d6534806a9061ffe5d15a434]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:16:19.450917 9131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c243f3145650481fb33e67bfde272cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:16:19.462427 9477 tablet_bootstrap.cc:492] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3: Bootstrap starting.
I20260929 02:16:19.465899 9477 tablet_bootstrap.cc:654] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3: Neither blocks nor log segments found. Creating new log.
I20260929 02:16:19.467136 9475 tablet_bootstrap.cc:492] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8: Bootstrap starting.
I20260929 02:16:19.467294 9477 log.cc:824] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:16:19.467775 9476 tablet_bootstrap.cc:492] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37: Bootstrap starting.
I20260929 02:16:19.470430 9477 tablet_bootstrap.cc:492] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3: No bootstrap required, opened a new log
I20260929 02:16:19.470736 9477 ts_tablet_manager.cc:1397] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260929 02:16:19.471453 9476 tablet_bootstrap.cc:654] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37: Neither blocks nor log segments found. Creating new log.
I20260929 02:16:19.472031 9475 tablet_bootstrap.cc:654] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8: Neither blocks nor log segments found. Creating new log.
I20260929 02:16:19.472791 9476 log.cc:824] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37: Log is configured to *not* fsync() on all Append() calls
I20260929 02:16:19.473344 9475 log.cc:824] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:16:19.475798 9476 tablet_bootstrap.cc:492] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37: No bootstrap required, opened a new log
I20260929 02:16:19.476084 9476 ts_tablet_manager.cc:1397] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 02:16:19.476423 9475 tablet_bootstrap.cc:492] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8: No bootstrap required, opened a new log
I20260929 02:16:19.476693 9475 ts_tablet_manager.cc:1397] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 02:16:19.480712 9477 raft_consensus.cc:348] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:19.481204 9477 raft_consensus.cc:374] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:16:19.481308 9477 raft_consensus.cc:729] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60cfdee59b004958a18f9a2736e6f5b3, State: Initialized, Role: FOLLOWER
I20260929 02:16:19.481945 9477 consensus_queue.cc:260] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:19.483893 9477 ts_tablet_manager.cc:1428] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260929 02:16:19.483829 9476 raft_consensus.cc:348] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:19.484649 9476 raft_consensus.cc:374] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:16:19.484776 9476 raft_consensus.cc:729] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174473aa2b4c4e269ac9dfdb2c95ec37, State: Initialized, Role: FOLLOWER
I20260929 02:16:19.485352 9476 consensus_queue.cc:260] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:19.486409 9459 heartbeater.cc:503] Master 127.24.216.190:43751 was elected leader, sending a full tablet report...
I20260929 02:16:19.488332 9476 ts_tablet_manager.cc:1428] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260929 02:16:19.488492 9475 raft_consensus.cc:348] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:19.488972 9475 raft_consensus.cc:374] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:16:19.489121 9475 raft_consensus.cc:729] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62b7e8e0afb24c948fdcc342df8b97c8, State: Initialized, Role: FOLLOWER
I20260929 02:16:19.489799 9475 consensus_queue.cc:260] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:19.490859 9328 heartbeater.cc:503] Master 127.24.216.190:43751 was elected leader, sending a full tablet report...
I20260929 02:16:19.491611 9475 ts_tablet_manager.cc:1428] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260929 02:16:19.534529 9481 raft_consensus.cc:482] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:16:19.534881 9481 raft_consensus.cc:504] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:19.536616 9481 leader_election.cc:288] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877), 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741)
I20260929 02:16:19.545810 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.537065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:59269 (local address 127.24.216.131:46741)
0929 02:16:19.537280 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:19.537309 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:19.538107 (+ 798us) server_negotiation.cc:407] Connection header received
0929 02:16:19.538325 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:19.538331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:19.538409 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:19.538505 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:16:19.539388 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.540505 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.543536 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.544766 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.544905 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.544975 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:19.545413 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 02:16:19.545543 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:16:19.545812 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.537333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:36143 (local address 127.24.216.130:44877)
0929 02:16:19.537583 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:19.537588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:19.538102 (+ 514us) server_negotiation.cc:407] Connection header received
0929 02:16:19.538325 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:19.538331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:19.538410 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:19.538505 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:16:19.539388 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.540510 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.543427 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.544647 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.544806 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.544893 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:19.545414 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 02:16:19.545543 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:16:19.546106 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.536976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:46741 (local address 127.24.216.129:59269)
0929 02:16:19.537955 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:19.537976 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:16:19.538169 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:16:19.538539 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:16:19.538551 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:16:19.538592 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:16:19.539225 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:19.539233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.540669 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.540676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:19.543398 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:19.543411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.544963 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.544968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:19.545202 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.545274 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 02:16:19.545374 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:16:19.545572 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":249,"spinlock_wait_cycles":55424,"thread_start_us":614,"threads_started":1}
I20260929 02:16:19.546119 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.537067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:44877 (local address 127.24.216.129:36143)
0929 02:16:19.537955 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:19.537982 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:16:19.538174 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:16:19.538539 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:16:19.538544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:16:19.538592 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:16:19.539202 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:19.539221 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.540671 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.540676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:19.543259 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:19.543277 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.544927 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.544936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:19.545198 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.545271 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 02:16:19.545376 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 02:16:19.545572 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":213,"spinlock_wait_cycles":14208,"thread_start_us":546,"threads_started":1}
I20260929 02:16:19.547349 9283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c243f3145650481fb33e67bfde272cb9" candidate_uuid: "60cfdee59b004958a18f9a2736e6f5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" is_pre_election: true
I20260929 02:16:19.547343 9414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c243f3145650481fb33e67bfde272cb9" candidate_uuid: "60cfdee59b004958a18f9a2736e6f5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" is_pre_election: true
I20260929 02:16:19.547906 9283 raft_consensus.cc:2454] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60cfdee59b004958a18f9a2736e6f5b3 in term 0.
I20260929 02:16:19.547906 9414 raft_consensus.cc:2454] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60cfdee59b004958a18f9a2736e6f5b3 in term 0.
I20260929 02:16:19.548593 9087 leader_election.cc:302] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60cfdee59b004958a18f9a2736e6f5b3, 62b7e8e0afb24c948fdcc342df8b97c8; no voters:
I20260929 02:16:19.549001 9481 raft_consensus.cc:2781] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:16:19.549130 9481 raft_consensus.cc:482] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:16:19.549275 9481 raft_consensus.cc:3037] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:16:19.551577 9481 raft_consensus.cc:504] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:19.552181 9481 leader_election.cc:288] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 1 election: Requested vote from peers 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877), 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741)
I20260929 02:16:19.552651 9283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c243f3145650481fb33e67bfde272cb9" candidate_uuid: "60cfdee59b004958a18f9a2736e6f5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62b7e8e0afb24c948fdcc342df8b97c8"
I20260929 02:16:19.552733 9414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c243f3145650481fb33e67bfde272cb9" candidate_uuid: "60cfdee59b004958a18f9a2736e6f5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37"
I20260929 02:16:19.552848 9283 raft_consensus.cc:3037] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:16:19.553005 9414 raft_consensus.cc:3037] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:16:19.555138 9283 raft_consensus.cc:2454] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60cfdee59b004958a18f9a2736e6f5b3 in term 1.
I20260929 02:16:19.555557 9087 leader_election.cc:302] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60cfdee59b004958a18f9a2736e6f5b3, 62b7e8e0afb24c948fdcc342df8b97c8; no voters:
I20260929 02:16:19.555601 9414 raft_consensus.cc:2454] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60cfdee59b004958a18f9a2736e6f5b3 in term 1.
I20260929 02:16:19.555861 9481 raft_consensus.cc:2781] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:16:19.556367 9481 raft_consensus.cc:686] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 LEADER]: Becoming Leader. State: Replica: 60cfdee59b004958a18f9a2736e6f5b3, State: Running, Role: LEADER
I20260929 02:16:19.556869 9481 consensus_queue.cc:237] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:19.561420 9010 catalog_manager.cc:5495] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60cfdee59b004958a18f9a2736e6f5b3 (127.24.216.129). New cstate: current_term: 1 leader_uuid: "60cfdee59b004958a18f9a2736e6f5b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } health_report { overall_health: UNKNOWN } } }
I20260929 02:16:19.585608 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.579963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48058 (local address 127.24.216.129:33141)
0929 02:16:19.580205 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:19.580210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:19.580230 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:16:19.580418 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:19.580422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:19.580519 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:19.580652 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:16:19.581016 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.581800 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.583340 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.583799 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.583902 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.584454 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:16:19.584981 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:16:19.585109 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:19.585307 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 02:16:19.585381 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260929 02:16:19.586020 9460 tablet.cc:2369] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:16:19.594051 9414 raft_consensus.cc:1261] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 1 FOLLOWER]: Refusing update from remote peer 60cfdee59b004958a18f9a2736e6f5b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:16:19.594110 9283 raft_consensus.cc:1261] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 1 FOLLOWER]: Refusing update from remote peer 60cfdee59b004958a18f9a2736e6f5b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:16:19.595096 9486 consensus_queue.cc:1035] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:16:19.595430 9481 consensus_queue.cc:1035] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:16:19.600557 9329 tablet.cc:2369] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:16:19.627087 9011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:16:19.627396 9011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:16:19.631265 9011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Sending DeleteTablet for 3 replicas of tablet c243f3145650481fb33e67bfde272cb9
I20260929 02:16:19.632551 9394 tablet_service.cc:1514] Processing DeleteTablet for tablet c243f3145650481fb33e67bfde272cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:16:19 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:60432
I20260929 02:16:19.632681 9263 tablet_service.cc:1514] Processing DeleteTablet for tablet c243f3145650481fb33e67bfde272cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:16:19 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:59830
I20260929 02:16:19.633167 9131 tablet_service.cc:1514] Processing DeleteTablet for tablet c243f3145650481fb33e67bfde272cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:16:19 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:48042
I20260929 02:16:19.634169 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 9048
I20260929 02:16:19.661085 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.635783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.24.216.129:33141)
0929 02:16:19.635950 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:19.635957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:19.635974 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:16:19.636040 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:19.636045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:19.636159 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:19.636257 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:19.638469 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.639754 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.642173 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.642879 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.643016 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.650301 (+ 7285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:19.650317 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:19.650339 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:19.650397 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:19.655005 (+ 4608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:19.656399 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:19.656409 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:19.656413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:19.656607 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:19.657731 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:19.657736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:19.657740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:19.658269 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:19.658563 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:19.660406 (+ 1843us) server_negotiation.cc:299] Negotiation successful
0929 02:16:19.660674 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:16:19.663340 9191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:16:19.665076 9499 tablet_replica.cc:330] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37: stopping tablet replica
I20260929 02:16:19.665663 9499 raft_consensus.cc:2229] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:16:19.666461 9499 raft_consensus.cc:2258] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:16:19.667781 9500 tablet_replica.cc:330] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8: stopping tablet replica
I20260929 02:16:19.668345 9500 raft_consensus.cc:2229] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:16:19.668901 9500 raft_consensus.cc:2258] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:16:19.669090 9499 ts_tablet_manager.cc:1905] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:16:19.671000 9500 ts_tablet_manager.cc:1905] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:16:19.674465 9499 ts_tablet_manager.cc:1918] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:16:19.674636 9499 log.cc:1196] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/c243f3145650481fb33e67bfde272cb9
I20260929 02:16:19.675158 9499 ts_tablet_manager.cc:1939] T c243f3145650481fb33e67bfde272cb9 P 174473aa2b4c4e269ac9dfdb2c95ec37: Deleting consensus metadata
I20260929 02:16:19.676721 8997 catalog_manager.cc:4841] TS 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741): tablet c243f3145650481fb33e67bfde272cb9 (table test-table [id=d5dcc955d6534806a9061ffe5d15a434]) successfully deleted
I20260929 02:16:19.678447 9500 ts_tablet_manager.cc:1918] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:16:19.678632 9500 log.cc:1196] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/c243f3145650481fb33e67bfde272cb9
I20260929 02:16:19.679193 9500 ts_tablet_manager.cc:1939] T c243f3145650481fb33e67bfde272cb9 P 62b7e8e0afb24c948fdcc342df8b97c8: Deleting consensus metadata
I20260929 02:16:19.681209 8998 catalog_manager.cc:4841] TS 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877): tablet c243f3145650481fb33e67bfde272cb9 (table test-table [id=d5dcc955d6534806a9061ffe5d15a434]) successfully deleted
I20260929 02:16:19.771798 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 9048
I20260929 02:16:19.771993 9501 tablet_replica.cc:330] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3: stopping tablet replica
I20260929 02:16:19.789788 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 9200
W20260929 02:16:19.789906 8998 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.216.129:33141 (error 108)
I20260929 02:16:19.792137 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.791596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:33141 (local address 127.0.0.1:48062)
0929 02:16:19.791775 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:19.791916 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 02:16:19.792414 8998 catalog_manager.cc:4584] TS 60cfdee59b004958a18f9a2736e6f5b3 (127.24.216.129:33141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c243f3145650481fb33e67bfde272cb9: Network error: Client connection negotiation failed: client connection to 127.24.216.129:33141: connect: Connection refused (error 111)
I20260929 02:16:19.808555 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.791092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.24.216.130:44877)
0929 02:16:19.791303 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:19.791307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:19.791346 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:16:19.791417 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:19.791432 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:19.791538 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:19.791646 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:19.793375 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.794296 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.795719 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.796106 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.796214 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.802661 (+ 6447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:19.802677 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:19.802693 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:19.802734 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:19.805927 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:19.806687 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:19.806694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:19.806697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:19.806841 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:19.807212 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:19.807217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:19.807219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:19.807626 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:19.807872 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:19.808136 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 02:16:19.808301 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:16:19.809213 9323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:16:19.846338 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.845745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:33141 (local address 127.0.0.1:48078)
0929 02:16:19.846005 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:19.846131 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 02:16:19.847541 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.847093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:33141 (local address 127.0.0.1:48086)
0929 02:16:19.847276 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:19.847383 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 02:16:19.908120 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.907574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:33141 (local address 127.0.0.1:48100)
0929 02:16:19.907856 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:19.907955 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260929 02:16:19.909217 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.908829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:33141 (local address 127.0.0.1:48116)
0929 02:16:19.909007 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:19.909104 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 02:16:19.910307 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 9200
I20260929 02:16:19.928997 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 9331
I20260929 02:16:19.943123 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.929799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.24.216.131:46741)
0929 02:16:19.929981 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:19.929986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:19.930004 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:16:19.930219 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:19.930224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:19.930294 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:19.930378 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:19.931693 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.932501 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.933767 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:19.934248 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:19.934345 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:19.937070 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:19.937087 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:19.937104 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:19.937146 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:19.940416 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:19.941098 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:19.941105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:19.941108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:19.941258 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:19.941626 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:19.941629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:19.941631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:19.941973 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:19.942275 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:19.942554 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 02:16:19.942768 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:16:19.943783 9454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:16:19.990455 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.989830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:33141 (local address 127.0.0.1:48118)
0929 02:16:19.990114 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:19.990308 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 02:16:19.991590 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:19.991180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:33141 (local address 127.0.0.1:48126)
0929 02:16:19.991379 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:19.991457 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260929 02:16:20.034103 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 9331
I20260929 02:16:20.051961 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 8978
I20260929 02:16:20.053159 9041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:16:20.166711 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:20.166102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:33141 (local address 127.0.0.1:48132)
0929 02:16:20.166375 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:20.166486 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 02:16:20.166765 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 8978
I20260929 02:16:40.189030 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:43751
--webserver_interface=127.24.216.190
--webserver_port=41797
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:43751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:16:40.526001 9507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:16:40.526376 9507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:16:40.526543 9507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:16:40.533953 9507 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:16:40.534101 9507 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:16:40.534232 9507 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:16:40.534350 9507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:16:40.534456 9507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:16:40.534601 9507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:16:40.534695 9507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:16:40.534783 9507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:16:40.545006 9507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:43751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:43751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=41797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:16:40.546461 9507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:16:40.547858 9507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:16:40.560777 9512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:40.560880 9515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:40.560902 9513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:16:40.561138 9507 server_base.cc:1034] running on GCE node
I20260929 02:16:40.562270 9507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:16:40.563628 9507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:16:40.564838 9507 hybrid_clock.cc:648] HybridClock initialized: now 1790648200564783 us; error 25 us; skew 500 ppm
I20260929 02:16:40.565137 9507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:16:40 dist-test-slave-9k0h krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648200, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:16:40.580098 9507 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:16:40.581651 9507 webserver.cc:460] Webserver started at http://127.24.216.190:41797/ using document root <none> and password file <none>
I20260929 02:16:40.582435 9507 fs_manager.cc:359] Metadata directory not provided
I20260929 02:16:40.582528 9507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:16:40.587836 9507 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 02:16:40.590642 9522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:40.592028 9507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:16:40.592171 9507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "f411ba5e13c44711a03b3e6fdcd0eb68"
format_stamp: "Formatted at 2026-09-29 02:15:54 on dist-test-slave-9k0h"
server_key: "6cef6ca937bbad388cc40e0d20bf4bf1"
server_key_iv: "d8d07c97bfe785ad797bdae21bb31f8a"
server_key_version: "encryptionkey@0"
I20260929 02:16:40.593103 9507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:16:40.631940 9507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:16:40.636402 9507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:16:40.636634 9507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:16:40.655016 9507 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:43751
I20260929 02:16:40.655048 9573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:43751 every 8 connection(s)
I20260929 02:16:40.656579 9507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:16:40.663406 9574 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:16:40.666278 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 9507
I20260929 02:16:40.667100 9574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Bootstrap starting.
I20260929 02:16:40.667408 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:33141
--local_ip_for_outbound_sockets=127.24.216.129
--tserver_master_addrs=127.24.216.190:43751
--webserver_port=36359
--webserver_interface=127.24.216.129
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:16:40.691156 9574 log.cc:824] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Log is configured to *not* fsync() on all Append() calls
I20260929 02:16:40.707921 9574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:16:40.708530 9574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Bootstrap complete.
I20260929 02:16:40.717975 9574 raft_consensus.cc:348] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } }
I20260929 02:16:40.719019 9574 raft_consensus.cc:729] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f411ba5e13c44711a03b3e6fdcd0eb68, State: Initialized, Role: FOLLOWER
I20260929 02:16:40.719659 9574 consensus_queue.cc:260] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } }
I20260929 02:16:40.719923 9574 raft_consensus.cc:388] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:16:40.720054 9574 raft_consensus.cc:482] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:16:40.720248 9574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 2 FOLLOWER]: Advancing to term 3
I20260929 02:16:40.724321 9574 raft_consensus.cc:504] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } }
I20260929 02:16:40.724802 9574 leader_election.cc:302] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f411ba5e13c44711a03b3e6fdcd0eb68; no voters:
I20260929 02:16:40.725471 9574 leader_election.cc:288] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 02:16:40.725626 9578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 02:16:40.726816 9578 raft_consensus.cc:686] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [term 3 LEADER]: Becoming Leader. State: Replica: f411ba5e13c44711a03b3e6fdcd0eb68, State: Running, Role: LEADER
I20260929 02:16:40.727494 9578 consensus_queue.cc:237] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } }
I20260929 02:16:40.727700 9574 sys_catalog.cc:564] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:16:40.730293 9580 sys_catalog.cc:455] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f411ba5e13c44711a03b3e6fdcd0eb68. Latest consensus state: current_term: 3 leader_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } } }
I20260929 02:16:40.730806 9580 sys_catalog.cc:458] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: This master's current role is: LEADER
I20260929 02:16:40.730719 9579 sys_catalog.cc:455] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f411ba5e13c44711a03b3e6fdcd0eb68" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 43751 } } }
I20260929 02:16:40.731221 9579 sys_catalog.cc:458] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68 [sys.catalog]: This master's current role is: LEADER
I20260929 02:16:40.747645 9584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:16:40.756691 9584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:16:40.758199 9584 catalog_manager.cc:1259] Loaded cluster ID: b1c75e5dfa434f0080908bb1ae4d127d
I20260929 02:16:40.758359 9584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:16:40.762624 9584 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:16:40.764369 9584 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Loaded TSK: 0
I20260929 02:16:40.764578 9584 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f411ba5e13c44711a03b3e6fdcd0eb68: Loaded TSK: 1
F20260929 02:16:40.765772 9584 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f50d2bab9ad google::LogMessage::Fail() at ??:0
@ 0x7f50d2bae88a google::LogMessage::SendToLog() at ??:0
@ 0x7f50d2bab4de google::LogMessage::Flush() at ??:0
@ 0x7f50d2baca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f50d690f7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f50d68fe73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f50e85da1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f50e85d9783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f50e86652a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f50e86650c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f50e86af5e9 std::function<>::operator()() at ??:0
@ 0x7f50e85ddcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f50e85dbc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f50e8663610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f50e8663439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f50d57f7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f50d57fa91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f50d57fa799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f50d57d5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f50d8aaf6db start_thread at ??:0
@ 0x7f50d1a4561f clone at ??:0
W20260929 02:16:41.073594 9576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:16:41.073923 9576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:16:41.074023 9576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:16:41.081410 9576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:16:41.081521 9576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:16:41.081634 9576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:16:41.091395 9576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:33141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=36359
--tserver_master_addrs=127.24.216.190:43751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:16:41.092638 9576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:16:41.093940 9576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:16:41.107100 9622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:41.107127 9625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:41.107172 9623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:16:41.108191 9576 server_base.cc:1034] running on GCE node
I20260929 02:16:41.108800 9576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:16:41.109712 9576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:16:41.110947 9576 hybrid_clock.cc:648] HybridClock initialized: now 1790648201110904 us; error 50 us; skew 500 ppm
I20260929 02:16:41.111299 9576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:16:41 dist-test-slave-9k0h krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648201, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:16:41.114598 9576 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:16:41.116087 9576 webserver.cc:460] Webserver started at http://127.24.216.129:36359/ using document root <none> and password file <none>
I20260929 02:16:41.116775 9576 fs_manager.cc:359] Metadata directory not provided
I20260929 02:16:41.116932 9576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:16:41.122610 9576 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:16:41.125481 9632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:41.126953 9576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:16:41.127134 9576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "60cfdee59b004958a18f9a2736e6f5b3"
format_stamp: "Formatted at 2026-09-29 02:15:55 on dist-test-slave-9k0h"
server_key: "01c381e5a6e9b02b3d19d7a6c4ddd2d3"
server_key_iv: "c5aff054dbdb2b34289dbf58d60c7ab4"
server_key_version: "encryptionkey@0"
I20260929 02:16:41.128080 9576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:16:41.149089 9576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:16:41.153118 9576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:16:41.153398 9576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:16:41.154455 9576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:16:41.156261 9639 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 02:16:41.161835 9576 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 02:16:41.162009 9576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 02:16:41.162173 9576 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 02:16:41.165449 9576 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 02:16:41.165524 9576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260929 02:16:41.165886 9639 tablet_bootstrap.cc:492] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3: Bootstrap starting.
I20260929 02:16:41.193060 9639 log.cc:824] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:16:41.205840 9576 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:33141
I20260929 02:16:41.205864 9746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:33141 every 8 connection(s)
I20260929 02:16:41.207387 9576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:16:41.209283 9639 tablet_bootstrap.cc:492] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:16:41.209908 9639 tablet_bootstrap.cc:492] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3: Bootstrap complete.
I20260929 02:16:41.211647 9639 ts_tablet_manager.cc:1397] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3: Time spent bootstrapping tablet: real 0.046s user 0.035s sys 0.009s
I20260929 02:16:41.212319 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:41.210301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:45713)
0929 02:16:41.211136 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:41.211381 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":390,"threads_started":1}
W20260929 02:16:41.212949 9747 heartbeater.cc:644] Failed to heartbeat to 127.24.216.190:43751 (0 consecutive failures): Network error: Failed to ping master at 127.24.216.190:43751: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
I20260929 02:16:41.213796 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:41.213478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:58033)
0929 02:16:41.213610 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:41.213671 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 02:16:41.214743 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:41.214359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:56587)
0929 02:16:41.214519 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:41.214607 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 02:16:41.214996 9747 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:16:41.218114 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 9576
I20260929 02:16:41.219087 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:44877
--local_ip_for_outbound_sockets=127.24.216.130
--tserver_master_addrs=127.24.216.190:43751
--webserver_port=34413
--webserver_interface=127.24.216.130
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:16:41.221155 9639 raft_consensus.cc:348] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:41.222085 9639 raft_consensus.cc:729] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60cfdee59b004958a18f9a2736e6f5b3, State: Initialized, Role: FOLLOWER
I20260929 02:16:41.222779 9639 consensus_queue.cc:260] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:41.226294 9639 ts_tablet_manager.cc:1428] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260929 02:16:41.226590 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:41.226061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:59305)
0929 02:16:41.226334 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:41.226432 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260929 02:16:41.227499 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:41.227166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:44521)
0929 02:16:41.227310 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:41.227387 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:16:41.544220 9751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:16:41.544538 9751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:16:41.544715 9751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:16:41.551720 9751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:16:41.551873 9751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:16:41.552063 9751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:16:41.561468 9751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:44877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=34413
--tserver_master_addrs=127.24.216.190:43751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:16:41.562785 9751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:16:41.564172 9751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:16:41.577718 9756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:41.577770 9757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:41.578183 9759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:16:41.578608 9751 server_base.cc:1034] running on GCE node
I20260929 02:16:41.579192 9751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:16:41.580107 9751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:16:41.581331 9751 hybrid_clock.cc:648] HybridClock initialized: now 1790648201581278 us; error 51 us; skew 500 ppm
I20260929 02:16:41.581627 9751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:16:41 dist-test-slave-9k0h krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648201, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:16:41.584707 9751 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:16:41.586135 9751 webserver.cc:460] Webserver started at http://127.24.216.130:34413/ using document root <none> and password file <none>
I20260929 02:16:41.586833 9751 fs_manager.cc:359] Metadata directory not provided
I20260929 02:16:41.586974 9751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:16:41.592033 9751 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:16:41.594616 9766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:41.595855 9751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:16:41.596019 9751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "62b7e8e0afb24c948fdcc342df8b97c8"
format_stamp: "Formatted at 2026-09-29 02:15:55 on dist-test-slave-9k0h"
server_key: "e25ecf2d133e2bf74b1c3e78e3e27e1d"
server_key_iv: "4841d34682e617b453a3ee1b686fbefa"
server_key_version: "encryptionkey@0"
I20260929 02:16:41.596936 9751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:16:41.609735 9751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:16:41.613977 9751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:16:41.614320 9751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:16:41.615255 9751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:16:41.616544 9751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:16:41.616662 9751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:41.616796 9751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:16:41.616890 9751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:41.649971 9751 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:44877
I20260929 02:16:41.650058 9878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:44877 every 8 connection(s)
I20260929 02:16:41.651471 9751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:16:41.656227 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:41.654469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:54259)
0929 02:16:41.655167 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:41.655392 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":258,"threads_started":1}
I20260929 02:16:41.656771 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 9751
W20260929 02:16:41.657012 9879 heartbeater.cc:644] Failed to heartbeat to 127.24.216.190:43751 (0 consecutive failures): Network error: Failed to ping master at 127.24.216.190:43751: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
I20260929 02:16:41.657532 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:46741
--local_ip_for_outbound_sockets=127.24.216.131
--tserver_master_addrs=127.24.216.190:43751
--webserver_port=44227
--webserver_interface=127.24.216.131
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:44327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:16:41.657935 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:41.657519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:33591)
0929 02:16:41.657694 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:41.657791 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 02:16:41.658849 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:41.658438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:40179)
0929 02:16:41.658596 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:41.658695 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 02:16:41.659103 9879 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:16:41.990608 9882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:16:41.990937 9882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:16:41.991119 9882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:16:41.998529 9882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:16:41.998677 9882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:16:41.998849 9882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:16:42.008967 9882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:44327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:46741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=44227
--tserver_master_addrs=127.24.216.190:43751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:16:42.010281 9882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:16:42.011582 9882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:16:42.024919 9888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:42.024919 9890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:42.025483 9887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:16:42.025875 9882 server_base.cc:1034] running on GCE node
I20260929 02:16:42.026500 9882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:16:42.027415 9882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:16:42.028621 9882 hybrid_clock.cc:648] HybridClock initialized: now 1790648202028563 us; error 51 us; skew 500 ppm
I20260929 02:16:42.028930 9882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:16:42 dist-test-slave-9k0h krb5kdc[8425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648202, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:16:42.032228 9882 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:16:42.033754 9882 webserver.cc:460] Webserver started at http://127.24.216.131:44227/ using document root <none> and password file <none>
I20260929 02:16:42.034485 9882 fs_manager.cc:359] Metadata directory not provided
I20260929 02:16:42.034638 9882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:16:42.039852 9882 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 02:16:42.042573 9897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:42.043916 9882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:16:42.044111 9882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "174473aa2b4c4e269ac9dfdb2c95ec37"
format_stamp: "Formatted at 2026-09-29 02:15:56 on dist-test-slave-9k0h"
server_key: "b6b0f921f36db77ee2b9fbf796716138"
server_key_iv: "b0540a9892aed15027b2ccbe46b65a36"
server_key_version: "encryptionkey@0"
I20260929 02:16:42.045115 9882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:16:42.058964 9882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:16:42.063338 9882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:16:42.063608 9882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:16:42.064622 9882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:16:42.065976 9882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:16:42.066066 9882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:42.066241 9882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:16:42.066306 9882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:42.100628 9882 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:46741
I20260929 02:16:42.100731 10009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:46741 every 8 connection(s)
I20260929 02:16:42.102193 9882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:16:42.104087 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 9882
I20260929 02:16:42.108084 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:42.105679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:45249)
0929 02:16:42.106986 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:42.107296 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":976,"thread_start_us":673,"threads_started":1}
W20260929 02:16:42.108711 10010 heartbeater.cc:644] Failed to heartbeat to 127.24.216.190:43751 (0 consecutive failures): Network error: Failed to ping master at 127.24.216.190:43751: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
I20260929 02:16:42.109474 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:42.109191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:49349)
0929 02:16:42.109313 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:42.109382 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 02:16:42.110347 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:42.109976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:45395)
0929 02:16:42.110113 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:42.110240 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 02:16:42.110581 10010 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:16:42.229167 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:42.228316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:51409)
0929 02:16:42.228855 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:42.228940 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":274,"threads_started":1}
I20260929 02:16:42.660497 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:42.659608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:56881)
0929 02:16:42.660117 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:42.660256 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":231,"threads_started":1}
I20260929 02:16:42.971952 10016 raft_consensus.cc:482] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:16:42.972281 10016 raft_consensus.cc:504] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:42.973794 10016 leader_election.cc:288] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877), 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741)
Sep 29 02:16:42 dist-test-slave-9k0h krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648201, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
Sep 29 02:16:42 dist-test-slave-9k0h krb5kdc[8425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648201, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:16:42.996627 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:42.974082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:46741 (local address 127.24.216.129:52095)
0929 02:16:42.975618 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:42.975653 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:16:42.976297 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:16:42.978790 (+ 2493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:16:42.978804 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:16:42.979544 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:16:42.980179 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:42.980199 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:42.981851 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:42.981855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:42.983343 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:42.983356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:42.984323 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:42.984329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:42.984590 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:42.985634 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:16:42.985683 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:16:42.988338 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:16:42.992820 (+ 4482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:16:42.992829 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:16:42.992847 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:16:42.993470 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:16:42.993926 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:16:42.993932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:16:42.993935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:16:42.994217 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:16:42.995006 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:16:42.995012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:16:42.995328 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 02:16:42.995593 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 02:16:42.996137 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1427,"mutex_wait_us":224,"spinlock_wait_cycles":57984,"thread_start_us":823,"threads_started":1}
I20260929 02:16:42.996616 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:42.974422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:52095 (local address 127.24.216.131:46741)
0929 02:16:42.975134 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:42.975153 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:42.975798 (+ 645us) server_negotiation.cc:407] Connection header received
0929 02:16:42.977932 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:42.977940 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:42.978356 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:42.978686 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:42.980387 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:42.981675 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:42.983493 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:42.984129 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:42.984316 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:42.988633 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:42.988642 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:42.988679 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:42.988727 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:42.992657 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:42.993637 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:42.993644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:42.993649 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:42.993808 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:42.994339 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:42.994343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:42.994345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:42.994853 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:42.995064 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:42.995814 (+ 750us) server_negotiation.cc:299] Negotiation successful
0929 02:16:42.996177 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":298,"threads_started":1}
I20260929 02:16:42.996629 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:42.974471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:34921 (local address 127.24.216.130:44877)
0929 02:16:42.976118 (+ 1647us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:42.976131 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:42.976174 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:16:42.979704 (+ 3530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:42.979712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:42.980050 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:42.980352 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:42.981292 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:42.982717 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:42.983980 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:42.984568 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:42.984755 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:42.988854 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:42.988865 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:42.988880 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:42.988921 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:42.992633 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:42.993633 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:42.993645 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:42.993651 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:42.993806 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:42.994327 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:42.994334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:42.994337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:42.994838 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:42.995038 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:42.995815 (+ 777us) server_negotiation.cc:299] Negotiation successful
0929 02:16:42.996204 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1467,"spinlock_wait_cycles":34688,"thread_start_us":1309,"threads_started":1}
I20260929 02:16:42.997033 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:42.974076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:44877 (local address 127.24.216.129:34921)
0929 02:16:42.975579 (+ 1503us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:42.975747 (+ 168us) client_negotiation.cc:174] Beginning negotiation
0929 02:16:42.976279 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:16:42.980319 (+ 4040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:16:42.980324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:16:42.980867 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:16:42.981123 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:42.981132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:42.982873 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:42.982885 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:42.983841 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:42.983852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:42.984767 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:42.984774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:42.984993 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:42.985828 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:16:42.985848 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:16:42.988594 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:16:42.992816 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:16:42.992829 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:16:42.992845 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:16:42.993472 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:16:42.993931 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:16:42.993936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:16:42.993946 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:16:42.994221 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:16:42.995001 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:16:42.995009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:16:42.995328 (+ 319us) client_negotiation.cc:769] Sending connection context
0929 02:16:42.995593 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 02:16:42.996338 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1184,"mutex_wait_us":244,"spinlock_wait_cycles":7040,"thread_start_us":1219,"threads_started":1}
W20260929 02:16:43.000051 9635 leader_election.cc:341] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741): Not found: Tablet not found: c243f3145650481fb33e67bfde272cb9
W20260929 02:16:43.000372 9636 leader_election.cc:341] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877): Not found: Tablet not found: c243f3145650481fb33e67bfde272cb9
I20260929 02:16:43.000537 9636 leader_election.cc:302] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60cfdee59b004958a18f9a2736e6f5b3; no voters: 174473aa2b4c4e269ac9dfdb2c95ec37, 62b7e8e0afb24c948fdcc342df8b97c8
I20260929 02:16:43.000954 10016 raft_consensus.cc:2726] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:16:43.111936 10021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:43.111070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:35359)
0929 02:16:43.111560 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:43.111699 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":262,"threads_started":1}
I20260929 02:16:43.230579 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:43.230123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:55857)
0929 02:16:43.230340 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:43.230440 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 02:16:43.662112 10022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:43.661365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:60499)
0929 02:16:43.661883 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:43.661960 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":287,"threads_started":1}
I20260929 02:16:44.113369 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:44.112727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:42661)
0929 02:16:44.113142 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:44.113222 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":209,"threads_started":1}
I20260929 02:16:44.232288 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:44.231523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:51169)
0929 02:16:44.232019 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:44.232103 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":208,"threads_started":1}
I20260929 02:16:44.663829 10025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:44.663016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:34659)
0929 02:16:44.663553 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:44.663643 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":249,"threads_started":1}
I20260929 02:16:44.727428 10026 raft_consensus.cc:482] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:16:44.727689 10026 raft_consensus.cc:504] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:44.728488 10026 leader_election.cc:288] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877), 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741)
W20260929 02:16:44.729503 9635 leader_election.cc:341] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741): Not found: Tablet not found: c243f3145650481fb33e67bfde272cb9
W20260929 02:16:44.729710 9636 leader_election.cc:341] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877): Not found: Tablet not found: c243f3145650481fb33e67bfde272cb9
I20260929 02:16:44.729825 9636 leader_election.cc:302] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60cfdee59b004958a18f9a2736e6f5b3; no voters: 174473aa2b4c4e269ac9dfdb2c95ec37, 62b7e8e0afb24c948fdcc342df8b97c8
I20260929 02:16:44.730041 10026 raft_consensus.cc:2726] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:16:45.114856 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:45.114118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:42753)
0929 02:16:45.114610 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:45.114687 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":259,"threads_started":1}
I20260929 02:16:45.233875 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:45.233078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:51545)
0929 02:16:45.233538 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:45.233648 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":225,"threads_started":1}
I20260929 02:16:45.665407 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:45.664720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:38181)
0929 02:16:45.665141 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:45.665223 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":212,"threads_started":1}
I20260929 02:16:46.116335 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:46.115632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:37789)
0929 02:16:46.116073 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:46.116150 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
I20260929 02:16:46.235764 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:46.235070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:34359)
0929 02:16:46.235518 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:46.235593 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":248,"threads_started":1}
I20260929 02:16:46.666944 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:46.666267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:50241)
0929 02:16:46.666713 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:46.666789 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":236,"threads_started":1}
I20260929 02:16:47.118031 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:47.117190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:52071)
0929 02:16:47.117789 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:47.117868 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":379,"threads_started":1}
I20260929 02:16:47.237391 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:47.236700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:33687)
0929 02:16:47.237129 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:47.237225 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":242,"threads_started":1}
I20260929 02:16:47.613773 10035 raft_consensus.cc:482] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:16:47.614080 10035 raft_consensus.cc:504] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:47.614791 10035 leader_election.cc:288] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877), 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741)
W20260929 02:16:47.615840 9635 leader_election.cc:341] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741): Not found: Tablet not found: c243f3145650481fb33e67bfde272cb9
W20260929 02:16:47.616060 9636 leader_election.cc:341] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877): Not found: Tablet not found: c243f3145650481fb33e67bfde272cb9
I20260929 02:16:47.616207 9636 leader_election.cc:302] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60cfdee59b004958a18f9a2736e6f5b3; no voters: 174473aa2b4c4e269ac9dfdb2c95ec37, 62b7e8e0afb24c948fdcc342df8b97c8
I20260929 02:16:47.616472 10035 raft_consensus.cc:2726] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:16:47.668565 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:47.667802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:41107)
0929 02:16:47.668285 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:47.668387 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":274,"threads_started":1}
I20260929 02:16:48.119854 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:48.118929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:51719)
0929 02:16:48.119452 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:48.119571 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":273,"threads_started":1}
I20260929 02:16:48.239063 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:48.238337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:49589)
0929 02:16:48.238779 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:48.238871 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":222,"threads_started":1}
I20260929 02:16:48.670461 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:48.669561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:32777)
0929 02:16:48.670075 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:48.670269 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":229,"threads_started":1}
I20260929 02:16:49.121523 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:49.120785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:35061)
0929 02:16:49.121261 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:49.121342 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":232,"threads_started":1}
I20260929 02:16:49.240597 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:49.239883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:50629)
0929 02:16:49.240339 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:49.240428 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":232,"threads_started":1}
I20260929 02:16:49.672024 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:49.671353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:56491)
0929 02:16:49.671768 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:49.671853 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":227,"threads_started":1}
I20260929 02:16:50.123208 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:50.122509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:53141)
0929 02:16:50.122920 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:50.123004 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":216,"threads_started":1}
I20260929 02:16:50.242265 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:50.241511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:48935)
0929 02:16:50.241942 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:50.242021 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":237,"threads_started":1}
I20260929 02:16:50.548547 10045 raft_consensus.cc:482] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:16:50.548804 10045 raft_consensus.cc:504] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:50.549486 10045 leader_election.cc:288] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877), 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741)
W20260929 02:16:50.550473 9636 leader_election.cc:341] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877): Not found: Tablet not found: c243f3145650481fb33e67bfde272cb9
W20260929 02:16:50.550657 9635 leader_election.cc:341] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741): Not found: Tablet not found: c243f3145650481fb33e67bfde272cb9
I20260929 02:16:50.550827 9635 leader_election.cc:302] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60cfdee59b004958a18f9a2736e6f5b3; no voters: 174473aa2b4c4e269ac9dfdb2c95ec37, 62b7e8e0afb24c948fdcc342df8b97c8
I20260929 02:16:50.551090 10045 raft_consensus.cc:2726] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:16:50.673719 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:50.673022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:58867)
0929 02:16:50.673482 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:50.673567 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":237,"threads_started":1}
I20260929 02:16:51.124809 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:51.124046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:40137)
0929 02:16:51.124576 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:51.124659 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":294,"threads_started":1}
I20260929 02:16:51.243993 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:51.243305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:58369)
0929 02:16:51.243764 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:51.243839 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":255,"threads_started":1}
I20260929 02:16:51.675264 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:51.674600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:60475)
0929 02:16:51.675011 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:51.675087 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
I20260929 02:16:52.126417 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:52.125674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:58707)
0929 02:16:52.126104 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:52.126241 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":220,"threads_started":1}
I20260929 02:16:52.245664 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:52.244918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:32791)
0929 02:16:52.245376 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:52.245477 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":244,"threads_started":1}
I20260929 02:16:52.676785 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:52.676126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:60683)
0929 02:16:52.676553 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:52.676632 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":208,"threads_started":1}
I20260929 02:16:53.127909 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:53.127258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:37197)
0929 02:16:53.127680 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:53.127758 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":207,"threads_started":1}
I20260929 02:16:53.247298 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:53.246580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:45871)
0929 02:16:53.247049 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:53.247136 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":250,"threads_started":1}
I20260929 02:16:53.678398 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:53.677619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:48287)
0929 02:16:53.678056 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:53.678141 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":230,"threads_started":1}
I20260929 02:16:54.031564 10056 raft_consensus.cc:482] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:16:54.031780 10056 raft_consensus.cc:504] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b7e8e0afb24c948fdcc342df8b97c8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44877 } } peers { permanent_uuid: "60cfdee59b004958a18f9a2736e6f5b3" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 33141 } } peers { permanent_uuid: "174473aa2b4c4e269ac9dfdb2c95ec37" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 46741 } }
I20260929 02:16:54.032550 10056 leader_election.cc:288] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877), 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741)
W20260929 02:16:54.033615 9636 leader_election.cc:341] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 62b7e8e0afb24c948fdcc342df8b97c8 (127.24.216.130:44877): Not found: Tablet not found: c243f3145650481fb33e67bfde272cb9
W20260929 02:16:54.033793 9635 leader_election.cc:341] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 174473aa2b4c4e269ac9dfdb2c95ec37 (127.24.216.131:46741): Not found: Tablet not found: c243f3145650481fb33e67bfde272cb9
I20260929 02:16:54.033954 9635 leader_election.cc:302] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60cfdee59b004958a18f9a2736e6f5b3; no voters: 174473aa2b4c4e269ac9dfdb2c95ec37, 62b7e8e0afb24c948fdcc342df8b97c8
I20260929 02:16:54.034278 10056 raft_consensus.cc:2726] T c243f3145650481fb33e67bfde272cb9 P 60cfdee59b004958a18f9a2736e6f5b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:16:54.129530 10057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:54.128807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:50437)
0929 02:16:54.129269 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:54.129357 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":250,"threads_started":1}
I20260929 02:16:54.248950 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:54.248150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:52295)
0929 02:16:54.248623 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:54.248729 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":234,"threads_started":1}
I20260929 02:16:54.679955 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:54.679248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:51461)
0929 02:16:54.679701 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:54.679800 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
I20260929 02:16:55.131206 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:55.130515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:46105)
0929 02:16:55.130950 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:55.131042 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":221,"threads_started":1}
I20260929 02:16:55.250769 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:55.249808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:41621)
0929 02:16:55.250323 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:55.250449 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":214,"threads_started":1}
I20260929 02:16:55.681825 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:55.680839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:37973)
0929 02:16:55.681510 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:55.681591 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":382,"threads_started":1}
I20260929 02:16:56.132860 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:56.132132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:45529)
0929 02:16:56.132576 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:56.132651 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":206,"threads_started":1}
I20260929 02:16:56.252615 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:56.251970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.129:36565)
0929 02:16:56.252370 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:56.252443 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":226,"threads_started":1}
I20260929 02:16:56.683699 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:56.682892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.130:46217)
0929 02:16:56.683406 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:56.683535 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":291,"threads_started":1}
I20260929 02:16:57.109225 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 9576
I20260929 02:16:57.124301 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:57.110380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.24.216.129:33141)
0929 02:16:57.110891 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:57.110902 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:57.110936 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:16:57.111011 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:57.111018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:57.111244 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:57.111432 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:57.113011 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:57.113962 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:57.115243 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:57.115805 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:57.115942 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:57.118613 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:57.118624 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:57.118638 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:57.118669 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:57.121408 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:57.122100 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:57.122107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:57.122110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:57.122294 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:57.122684 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:57.122689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:57.122693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:57.123060 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:57.123376 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:57.123677 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 02:16:57.123965 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"spinlock_wait_cycles":3328,"thread_start_us":317,"threads_started":1}
I20260929 02:16:57.125520 9742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:16:57.134806 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:57.134021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:43751 (local address 127.24.216.131:45313)
0929 02:16:57.134533 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:57.134656 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.190:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":291,"threads_started":1}
I20260929 02:16:57.219336 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 9576
I20260929 02:16:57.238793 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 9751
I20260929 02:16:57.252240 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:57.239523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.24.216.130:44877)
0929 02:16:57.240010 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:57.240015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:57.240029 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:16:57.240082 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:57.240086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:57.240142 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:57.240273 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:57.241624 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:57.242437 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:57.243726 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:57.244170 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:57.244318 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:57.247015 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:57.247020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:57.247024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:57.247056 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:57.249906 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:57.250621 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:57.250626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:57.250629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:57.250767 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:57.251075 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:57.251078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:57.251080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:57.251340 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:57.251533 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:57.251788 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 02:16:57.251956 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":251,"threads_started":1}
I20260929 02:16:57.252887 9874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:16:57.336007 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 9751
I20260929 02:16:57.353797 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 9882
I20260929 02:16:57.366832 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:57.354646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.24.216.131:46741)
0929 02:16:57.355170 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:57.355175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:57.355192 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:16:57.355249 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:57.355253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:57.355320 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:57.355413 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:57.356694 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:57.357446 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:57.358750 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:57.359183 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:57.359297 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:57.361815 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:57.361821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:57.361824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:57.361857 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:57.364394 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:57.365052 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:57.365056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:57.365058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:57.365183 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:57.365488 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:57.365491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:57.365493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:57.365772 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:57.365969 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:57.366258 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 02:16:57.366426 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":258,"threads_started":1}
I20260929 02:16:57.367539 10005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:16:57.452479 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 9882
2026-09-29T02:16:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64457 ms)
[----------] 30 tests from SecurityITest (296735 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:16:57 dist-test-slave-9k0h krb5kdc[10075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:16:57 dist-test-slave-9k0h krb5kdc[10075](info): set up 2 sockets
Sep 29 02:16:57 dist-test-slave-9k0h krb5kdc[10075](info): commencing operation
krb5kdc: starting...
W20260929 02:16:58.594449 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.012s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:16:58 dist-test-slave-9k0h krb5kdc[10075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:16:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:16:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:16:58.854025 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:36965
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:36965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:16:59.177929 10088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:16:59.178282 10088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:16:59.178447 10088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:16:59.185467 10088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:16:59.185606 10088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:16:59.185724 10088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:16:59.185832 10088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:16:59.185916 10088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:16:59.195555 10088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:36965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:36965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:16:59.196815 10088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:16:59.198088 10088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:16:59.210417 10093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:59.210383 10094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:59.210383 10096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:16:59.211694 10088 server_base.cc:1034] running on GCE node
I20260929 02:16:59.212370 10088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:16:59.213640 10088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:16:59.214869 10088 hybrid_clock.cc:648] HybridClock initialized: now 1790648219214837 us; error 50 us; skew 500 ppm
I20260929 02:16:59.215328 10088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:16:59 dist-test-slave-9k0h krb5kdc[10075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648219, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:16:59.223115 10088 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:16:59.224476 10088 webserver.cc:460] Webserver started at http://127.24.216.190:46765/ using document root <none> and password file <none>
I20260929 02:16:59.225104 10088 fs_manager.cc:359] Metadata directory not provided
I20260929 02:16:59.225209 10088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:16:59.225467 10088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:16:59.228412 10088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "d943decade394ee3a90438f7c8f7c560"
format_stamp: "Formatted at 2026-09-29 02:16:59 on dist-test-slave-9k0h"
server_key: "e7852eed74b836a69084fe9e4e9dbdc5"
server_key_iv: "02422d8dff67621c1a158325b1548f4d"
server_key_version: "encryptionkey@0"
I20260929 02:16:59.229094 10088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "d943decade394ee3a90438f7c8f7c560"
format_stamp: "Formatted at 2026-09-29 02:16:59 on dist-test-slave-9k0h"
server_key: "e7852eed74b836a69084fe9e4e9dbdc5"
server_key_iv: "02422d8dff67621c1a158325b1548f4d"
server_key_version: "encryptionkey@0"
I20260929 02:16:59.234021 10088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:16:59.237322 10103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:59.238709 10088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:16:59.238893 10088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "d943decade394ee3a90438f7c8f7c560"
format_stamp: "Formatted at 2026-09-29 02:16:59 on dist-test-slave-9k0h"
server_key: "e7852eed74b836a69084fe9e4e9dbdc5"
server_key_iv: "02422d8dff67621c1a158325b1548f4d"
server_key_version: "encryptionkey@0"
I20260929 02:16:59.239086 10088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:16:59.252856 10088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:16:59.257187 10088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:16:59.257483 10088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:16:59.275648 10088 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:36965
I20260929 02:16:59.275666 10154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:36965 every 8 connection(s)
I20260929 02:16:59.277021 10088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:16:59.279019 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 10088
I20260929 02:16:59.279268 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:16:59.280004 25442 external_mini_cluster.cc:1428] Setting key cdaf04c75e921c8cbaaed4b464b797ef
I20260929 02:16:59.280884 10155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:16:59.294618 10155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560: Bootstrap starting.
Sep 29 02:16:59 dist-test-slave-9k0h krb5kdc[10075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:16:59.297991 10155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560: Neither blocks nor log segments found. Creating new log.
I20260929 02:16:59.299360 10155 log.cc:824] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560: Log is configured to *not* fsync() on all Append() calls
I20260929 02:16:59.303241 10155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560: No bootstrap required, opened a new log
I20260929 02:16:59.304840 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:59.282204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39546 (local address 127.24.216.190:36965)
0929 02:16:59.283003 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:59.283018 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:59.283066 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:16:59.285459 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:59.285487 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:59.285977 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:59.286452 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:59.287939 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:59.289855 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:59.291351 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:59.292406 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:59.292588 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:59.296675 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:59.296684 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:59.296701 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:59.296754 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:59.300532 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:59.301236 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:59.301243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:59.301251 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:59.301452 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:59.301822 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:59.301826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:59.301829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:59.302515 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:59.302801 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:59.303149 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 02:16:59.303562 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":337,"threads_started":1}
I20260929 02:16:59.311383 10155 raft_consensus.cc:348] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d943decade394ee3a90438f7c8f7c560" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36965 } }
I20260929 02:16:59.311702 10155 raft_consensus.cc:374] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:16:59.311772 10155 raft_consensus.cc:729] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d943decade394ee3a90438f7c8f7c560, State: Initialized, Role: FOLLOWER
I20260929 02:16:59.312211 10155 consensus_queue.cc:260] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d943decade394ee3a90438f7c8f7c560" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36965 } }
I20260929 02:16:59.312394 10155 raft_consensus.cc:388] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:16:59.312482 10155 raft_consensus.cc:482] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:16:59.312597 10155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:16:59.314713 10155 raft_consensus.cc:504] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d943decade394ee3a90438f7c8f7c560" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36965 } }
I20260929 02:16:59.315114 10155 leader_election.cc:302] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d943decade394ee3a90438f7c8f7c560; no voters:
I20260929 02:16:59.315799 10155 leader_election.cc:288] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:16:59.315903 10160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:16:59.316669 10160 raft_consensus.cc:686] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [term 1 LEADER]: Becoming Leader. State: Replica: d943decade394ee3a90438f7c8f7c560, State: Running, Role: LEADER
I20260929 02:16:59.317207 10160 consensus_queue.cc:237] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d943decade394ee3a90438f7c8f7c560" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36965 } }
I20260929 02:16:59.317859 10155 sys_catalog.cc:564] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:16:59.319968 10162 sys_catalog.cc:455] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d943decade394ee3a90438f7c8f7c560. Latest consensus state: current_term: 1 leader_uuid: "d943decade394ee3a90438f7c8f7c560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d943decade394ee3a90438f7c8f7c560" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36965 } } }
I20260929 02:16:59.320310 10162 sys_catalog.cc:458] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [sys.catalog]: This master's current role is: LEADER
I20260929 02:16:59.320822 10161 sys_catalog.cc:455] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d943decade394ee3a90438f7c8f7c560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d943decade394ee3a90438f7c8f7c560" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 36965 } } }
I20260929 02:16:59.321128 10161 sys_catalog.cc:458] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560 [sys.catalog]: This master's current role is: LEADER
I20260929 02:16:59.322032 10166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:16:59.327116 10166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:16:59.335534 10166 catalog_manager.cc:1347] Generated new cluster ID: 40e07e7ca1ba480eb17e2a5c5df63cc0
I20260929 02:16:59.335652 10166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:16:59.360836 10166 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:16:59.362231 10166 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:16:59.369638 10166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560: Generated new TSK 0
I20260929 02:16:59.370517 10166 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:16:59.437719 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36965
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:16:59.763033 10181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:16:59.763334 10181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:16:59.763499 10181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:16:59.770479 10181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:16:59.770623 10181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:16:59.770810 10181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:16:59.780411 10181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:16:59.781749 10181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:16:59.783089 10181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:16:59.796103 10189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:59.796068 10186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:16:59.796068 10187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:16:59.796936 10181 server_base.cc:1034] running on GCE node
I20260929 02:16:59.797600 10181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:16:59.798596 10181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:16:59.799814 10181 hybrid_clock.cc:648] HybridClock initialized: now 1790648219799778 us; error 25 us; skew 500 ppm
I20260929 02:16:59.800135 10181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:16:59 dist-test-slave-9k0h krb5kdc[10075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648219, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:16:59.803501 10181 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:16:59.805145 10181 webserver.cc:460] Webserver started at http://127.24.216.129:42843/ using document root <none> and password file <none>
I20260929 02:16:59.805863 10181 fs_manager.cc:359] Metadata directory not provided
I20260929 02:16:59.806012 10181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:16:59.806419 10181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:16:59.809537 10181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "6f9e618c69be4d86828c69d3316cd5b1"
format_stamp: "Formatted at 2026-09-29 02:16:59 on dist-test-slave-9k0h"
server_key: "02a340e3ad54aca84231b609baa46e58"
server_key_iv: "c8692be43a0e51191422c19e95a84fac"
server_key_version: "encryptionkey@0"
I20260929 02:16:59.810477 10181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "6f9e618c69be4d86828c69d3316cd5b1"
format_stamp: "Formatted at 2026-09-29 02:16:59 on dist-test-slave-9k0h"
server_key: "02a340e3ad54aca84231b609baa46e58"
server_key_iv: "c8692be43a0e51191422c19e95a84fac"
server_key_version: "encryptionkey@0"
I20260929 02:16:59.816023 10181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:16:59.819916 10196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:59.821200 10181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:16:59.821384 10181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "6f9e618c69be4d86828c69d3316cd5b1"
format_stamp: "Formatted at 2026-09-29 02:16:59 on dist-test-slave-9k0h"
server_key: "02a340e3ad54aca84231b609baa46e58"
server_key_iv: "c8692be43a0e51191422c19e95a84fac"
server_key_version: "encryptionkey@0"
I20260929 02:16:59.821604 10181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:16:59.855466 10181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:16:59.860288 10181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:16:59.860548 10181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:16:59.861481 10181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:16:59.862795 10181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:16:59.862869 10181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:59.862957 10181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:16:59.863008 10181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:16:59.896836 10181 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:45033
I20260929 02:16:59.896885 10308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:45033 every 8 connection(s)
I20260929 02:16:59.898187 10181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:16:59.906208 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 10181
I20260929 02:16:59.906437 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:16:59.907262 25442 external_mini_cluster.cc:1428] Setting key 28896ac9877e8682681b9c23908e4472
Sep 29 02:16:59 dist-test-slave-9k0h krb5kdc[10075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648219, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:16:59.918520 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:59.900728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:41105 (local address 127.24.216.190:36965)
0929 02:16:59.901194 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 02:16:59.901200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:16:59.902100 (+ 900us) server_negotiation.cc:407] Connection header received
0929 02:16:59.903940 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:16:59.903944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:16:59.904029 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:16:59.904139 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:16:59.905564 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:59.906449 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:59.908617 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:16:59.909053 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:16:59.909202 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:59.912917 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:16:59.912922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:16:59.912925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:16:59.912955 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:16:59.915472 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:59.916168 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:59.916175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:59.916181 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:59.916304 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:16:59.916730 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:16:59.916741 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:16:59.916744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:16:59.916995 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:16:59.917150 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:16:59.917898 (+ 748us) server_negotiation.cc:299] Negotiation successful
0929 02:16:59.918117 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":203,"threads_started":1}
I20260929 02:16:59.919560 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:16:59.901279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36965 (local address 127.24.216.129:41105)
0929 02:16:59.901932 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 02:16:59.901974 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:16:59.903654 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:16:59.904340 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:16:59.904348 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:16:59.904862 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:16:59.905409 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:59.905424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:59.906623 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:59.906629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:59.908477 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:16:59.908489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:16:59.909286 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:16:59.909295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:16:59.909535 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:16:59.910275 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:16:59.910296 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:16:59.912707 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:16:59.915615 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:16:59.915624 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:16:59.915640 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:16:59.916045 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:16:59.916413 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:16:59.916418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:16:59.916421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:16:59.916601 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:16:59.917120 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:16:59.917127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:16:59.917463 (+ 336us) client_negotiation.cc:769] Sending connection context
0929 02:16:59.917794 (+ 331us) client_negotiation.cc:240] Negotiation successful
0929 02:16:59.918362 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":260,"threads_started":1}
I20260929 02:16:59.921144 10309 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36965
I20260929 02:16:59.921447 10309 heartbeater.cc:464] Registering TS with master...
I20260929 02:16:59.922200 10309 heartbeater.cc:511] Master 127.24.216.190:36965 requested a full tablet report, sending...
I20260929 02:16:59.924319 10120 ts_manager.cc:194] Registered new tserver with Master: 6f9e618c69be4d86828c69d3316cd5b1 (127.24.216.129:45033)
I20260929 02:16:59.925865 10120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:41105
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:16:59.967089 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36965
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:00.300645 10315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:00.300925 10315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:00.301023 10315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:00.308929 10315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:00.309062 10315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:00.309178 10315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:17:00.319881 10315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:00.321228 10315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:00.322659 10315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:00.335999 10320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:00.336076 10321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:00.336035 10323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:00.337074 10315 server_base.cc:1034] running on GCE node
I20260929 02:17:00.337659 10315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:00.338620 10315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:00.339836 10315 hybrid_clock.cc:648] HybridClock initialized: now 1790648220339794 us; error 52 us; skew 500 ppm
I20260929 02:17:00.340133 10315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:00 dist-test-slave-9k0h krb5kdc[10075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:00.343286 10315 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:17:00.344651 10315 webserver.cc:460] Webserver started at http://127.24.216.130:38607/ using document root <none> and password file <none>
I20260929 02:17:00.345306 10315 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:00.345440 10315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:00.345731 10315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:00.348572 10315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "1e53f686e1944c039f319d9cf068822c"
format_stamp: "Formatted at 2026-09-29 02:17:00 on dist-test-slave-9k0h"
server_key: "cfff2caca826eff97cc85162a77289e9"
server_key_iv: "99d3eca57f2c5b41d31356910a7f6505"
server_key_version: "encryptionkey@0"
I20260929 02:17:00.349337 10315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "1e53f686e1944c039f319d9cf068822c"
format_stamp: "Formatted at 2026-09-29 02:17:00 on dist-test-slave-9k0h"
server_key: "cfff2caca826eff97cc85162a77289e9"
server_key_iv: "99d3eca57f2c5b41d31356910a7f6505"
server_key_version: "encryptionkey@0"
I20260929 02:17:00.354298 10315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:17:00.357555 10330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:00.358832 10315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:17:00.358997 10315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "1e53f686e1944c039f319d9cf068822c"
format_stamp: "Formatted at 2026-09-29 02:17:00 on dist-test-slave-9k0h"
server_key: "cfff2caca826eff97cc85162a77289e9"
server_key_iv: "99d3eca57f2c5b41d31356910a7f6505"
server_key_version: "encryptionkey@0"
I20260929 02:17:00.359184 10315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:00.378415 10315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:00.382076 10315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:00.382400 10315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:00.383283 10315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:00.384545 10315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:00.384645 10315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:00.384783 10315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:00.384850 10315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:00.419610 10315 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:44565
I20260929 02:17:00.419667 10442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:44565 every 8 connection(s)
I20260929 02:17:00.421034 10315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:17:00.422554 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 10315
I20260929 02:17:00.422730 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:17:00.423213 25442 external_mini_cluster.cc:1428] Setting key e5d50686820cc5d356e27b488d58a3c3
Sep 29 02:17:00 dist-test-slave-9k0h krb5kdc[10075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:17:00.444321 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:00.425256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:40141 (local address 127.24.216.190:36965)
0929 02:17:00.425756 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:00.425762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:00.426983 (+ 1221us) server_negotiation.cc:407] Connection header received
0929 02:17:00.428824 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:00.428832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:00.428942 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:00.429060 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:00.430599 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:00.431794 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:00.433296 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:00.434048 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:00.434216 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:00.438340 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:00.438344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:00.438348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:00.438379 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:00.441085 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:00.441838 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:00.441848 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:00.441850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:00.442037 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:00.442493 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:00.442498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:00.442502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:00.442785 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:00.442953 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:00.443703 (+ 750us) server_negotiation.cc:299] Negotiation successful
0929 02:17:00.443939 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":272,"threads_started":1}
I20260929 02:17:00.445233 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:00.425965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36965 (local address 127.24.216.130:40141)
0929 02:17:00.426811 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:00.426862 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:00.428552 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:00.429282 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:00.429290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:00.429816 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:00.430436 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:00.430454 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:00.431937 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:00.431941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:00.433130 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:00.433140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:00.434293 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:00.434298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:00.434486 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:00.435262 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:00.435309 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:00.438083 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:00.441258 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:00.441267 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:00.441283 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:00.441709 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:00.442173 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:00.442178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:00.442182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:00.442373 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:00.442927 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:00.442935 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:00.443343 (+ 408us) client_negotiation.cc:769] Sending connection context
0929 02:17:00.443605 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 02:17:00.444135 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":385,"threads_started":1}
I20260929 02:17:00.446727 10443 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36965
I20260929 02:17:00.447021 10443 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:00.447731 10443 heartbeater.cc:511] Master 127.24.216.190:36965 requested a full tablet report, sending...
I20260929 02:17:00.449190 10120 ts_manager.cc:194] Registered new tserver with Master: 1e53f686e1944c039f319d9cf068822c (127.24.216.130:44565)
I20260929 02:17:00.450008 10120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:40141
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:17:00.485140 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36965
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:00.810441 10449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:00.810690 10449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:00.810835 10449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:00.818107 10449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:00.818214 10449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:00.818403 10449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:17:00.828153 10449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:36965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:00.829427 10449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:00.830780 10449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:00.843474 10454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:00.843708 10455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:00.844074 10457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:00.844741 10449 server_base.cc:1034] running on GCE node
I20260929 02:17:00.845299 10449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:00.846258 10449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:00.847465 10449 hybrid_clock.cc:648] HybridClock initialized: now 1790648220847406 us; error 49 us; skew 500 ppm
I20260929 02:17:00.847782 10449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:00 dist-test-slave-9k0h krb5kdc[10075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:00.850905 10449 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:17:00.852319 10449 webserver.cc:460] Webserver started at http://127.24.216.131:36915/ using document root <none> and password file <none>
I20260929 02:17:00.852972 10449 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:00.853104 10449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:00.853403 10449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:00.856251 10449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "8a279fc31af941bfa83575faddaa9803"
format_stamp: "Formatted at 2026-09-29 02:17:00 on dist-test-slave-9k0h"
server_key: "8c5546f143589b5fb4d486196b793a3f"
server_key_iv: "11cfc3a91a254bbb4c4a2f36f0b42d4d"
server_key_version: "encryptionkey@0"
I20260929 02:17:00.857064 10449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "8a279fc31af941bfa83575faddaa9803"
format_stamp: "Formatted at 2026-09-29 02:17:00 on dist-test-slave-9k0h"
server_key: "8c5546f143589b5fb4d486196b793a3f"
server_key_iv: "11cfc3a91a254bbb4c4a2f36f0b42d4d"
server_key_version: "encryptionkey@0"
I20260929 02:17:00.861939 10449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260929 02:17:00.865231 10464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:00.866483 10449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 02:17:00.866667 10449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "8a279fc31af941bfa83575faddaa9803"
format_stamp: "Formatted at 2026-09-29 02:17:00 on dist-test-slave-9k0h"
server_key: "8c5546f143589b5fb4d486196b793a3f"
server_key_iv: "11cfc3a91a254bbb4c4a2f36f0b42d4d"
server_key_version: "encryptionkey@0"
I20260929 02:17:00.866844 10449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:00.888159 10449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:00.892369 10449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:00.892659 10449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:00.893589 10449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:00.894860 10449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:00.894936 10449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:00.895103 10449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:00.895153 10449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:00.928478 10449 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:35617
I20260929 02:17:00.928530 10576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:35617 every 8 connection(s)
I20260929 02:17:00.929491 10309 heartbeater.cc:503] Master 127.24.216.190:36965 was elected leader, sending a full tablet report...
I20260929 02:17:00.929945 10449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:17:00.939383 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 10449
I20260929 02:17:00.939563 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:17:00.939988 25442 external_mini_cluster.cc:1428] Setting key a67f6cdb6972b1759efeac3341531015
Sep 29 02:17:00 dist-test-slave-9k0h krb5kdc[10075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:17:00.949465 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:00.932742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:51243 (local address 127.24.216.190:36965)
0929 02:17:00.932904 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:00.932909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:00.933994 (+ 1085us) server_negotiation.cc:407] Connection header received
0929 02:17:00.935838 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:00.935843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:00.935928 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:00.936045 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:00.937435 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:00.938276 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:00.939823 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:00.940317 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:00.940445 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:00.943823 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:00.943829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:00.943834 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:00.943878 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:00.946629 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:00.947295 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:00.947305 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:00.947307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:00.947423 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:00.947838 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:00.947844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:00.947847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:00.948102 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:00.948282 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:00.948912 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 02:17:00.949128 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:17:00.950419 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:00.933182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:36965 (local address 127.24.216.131:51243)
0929 02:17:00.933851 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:00.933886 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:00.935575 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:00.936273 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:00.936280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:00.936762 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:00.937279 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:00.937295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:00.938409 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:00.938415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:00.939678 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:00.939691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:00.940530 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:00.940537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:00.940770 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:00.941425 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:00.941451 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:00.943625 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:00.946760 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:00.946769 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:00.946783 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:00.947169 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:00.947528 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:00.947532 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:00.947536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:00.947730 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:00.948224 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:00.948232 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:00.948559 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 02:17:00.948821 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 02:17:00.949314 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":252,"threads_started":1}
I20260929 02:17:00.951689 10577 heartbeater.cc:347] Connected to a master server at 127.24.216.190:36965
I20260929 02:17:00.951951 10577 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:00.952561 10577 heartbeater.cc:511] Master 127.24.216.190:36965 requested a full tablet report, sending...
I20260929 02:17:00.954016 10119 ts_manager.cc:194] Registered new tserver with Master: 8a279fc31af941bfa83575faddaa9803 (127.24.216.131:35617)
I20260929 02:17:00.954959 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:17:00.955154 10119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:51243
I20260929 02:17:00.973951 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:00.960115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.24.216.190:36965)
0929 02:17:00.960317 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:00.960325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:00.960732 (+ 407us) server_negotiation.cc:407] Connection header received
0929 02:17:00.960920 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:00.960925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:00.961000 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:00.961099 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:00.962614 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:00.963332 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:00.964718 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:00.965145 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:00.965284 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:00.967812 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:00.967817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:00.967820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:00.967850 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:00.970850 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:00.971720 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:00.971731 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:00.971734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:00.971875 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:00.972275 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:00.972284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:00.972288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:00.972626 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:00.972927 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:00.973306 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 02:17:00.973547 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:17:00.978309 10119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:17:00.985239 10119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:17:01.022614 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:00.997490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50362 (local address 127.24.216.131:35617)
0929 02:17:01.008289 (+ 10799us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:01.008298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:01.013496 (+ 5198us) server_negotiation.cc:407] Connection header received
0929 02:17:01.013636 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:01.013644 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:01.013977 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:01.014249 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:01.015302 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.016656 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.020043 (+ 3387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.021491 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.021678 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:01.021828 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:01.022082 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 02:17:01.022279 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10612,"thread_start_us":10445,"threads_started":1}
I20260929 02:17:01.022382 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:00.997371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:45033 (local address 127.0.0.1:34404)
0929 02:17:01.014550 (+ 17179us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:01.014567 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:01.014720 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:01.015585 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:01.015590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:01.015617 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:01.015948 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:01.015959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.017556 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.017565 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:01.019698 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:01.019711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.021338 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.021343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:01.021474 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:01.021504 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:17:01.021642 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 02:17:01.021847 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17044,"mutex_wait_us":32,"spinlock_wait_cycles":12672,"thread_start_us":8932,"threads_started":1}
I20260929 02:17:01.023077 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:00.997994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.24.216.129:45033)
0929 02:17:01.012369 (+ 14375us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:01.012382 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:01.014682 (+ 2300us) server_negotiation.cc:407] Connection header received
0929 02:17:01.014953 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:01.014962 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:01.015208 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:01.015345 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:01.016116 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.017426 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.019904 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.020926 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.021060 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:01.021181 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:01.022510 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0929 02:17:01.022676 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14138,"thread_start_us":14398,"threads_started":1}
I20260929 02:17:01.023118 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:00.996987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.24.216.130:44565)
0929 02:17:01.013547 (+ 16560us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:01.013556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:01.013587 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:17:01.013682 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:01.013702 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:01.013955 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:01.014846 (+ 891us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:01.015534 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.016906 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.020043 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.021493 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.021649 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:01.021819 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:01.022600 (+ 781us) server_negotiation.cc:299] Negotiation successful
0929 02:17:01.022774 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16363,"thread_start_us":17182,"threads_started":1}
I20260929 02:17:01.022589 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:00.996976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:35617 (local address 127.0.0.1:50362)
0929 02:17:01.013093 (+ 16117us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:01.013120 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:01.013311 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:01.014483 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:01.014491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:01.014545 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:01.015044 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:01.015058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.016792 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.016800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:01.019856 (+ 3056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:01.019871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.021709 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.021715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:01.021881 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:01.021911 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:17:01.022003 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 02:17:01.022128 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15948,"mutex_wait_us":223,"thread_start_us":17291,"threads_started":1}
I20260929 02:17:01.023384 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:00.996811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:44565 (local address 127.0.0.1:39684)
0929 02:17:01.007183 (+ 10372us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:01.007241 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:01.007418 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:01.014834 (+ 7416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:01.014839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:01.014866 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:01.015356 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:01.015370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.017035 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.017040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:01.019856 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:01.019871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.021963 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.021994 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:01.022204 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:01.022257 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 02:17:01.022349 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 02:17:01.022495 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9565,"mutex_wait_us":146,"thread_start_us":16050,"threads_started":1}
I20260929 02:17:01.027360 10512 tablet_service.cc:1467] Processing CreateTablet for tablet a867141515e044f8b6b0f0364e583612 (DEFAULT_TABLE table=test-table [id=3e4d354c36ba4414837317d84455aa28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:01.028625 10512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a867141515e044f8b6b0f0364e583612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:01.028373 10378 tablet_service.cc:1467] Processing CreateTablet for tablet a867141515e044f8b6b0f0364e583612 (DEFAULT_TABLE table=test-table [id=3e4d354c36ba4414837317d84455aa28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:01.029440 10378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a867141515e044f8b6b0f0364e583612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:01.037868 10244 tablet_service.cc:1467] Processing CreateTablet for tablet a867141515e044f8b6b0f0364e583612 (DEFAULT_TABLE table=test-table [id=3e4d354c36ba4414837317d84455aa28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:01.039391 10244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a867141515e044f8b6b0f0364e583612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:01.040036 10595 tablet_bootstrap.cc:492] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803: Bootstrap starting.
I20260929 02:17:01.042923 10596 tablet_bootstrap.cc:492] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c: Bootstrap starting.
I20260929 02:17:01.043702 10595 tablet_bootstrap.cc:654] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:01.045015 10595 log.cc:824] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:01.046790 10596 tablet_bootstrap.cc:654] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:01.048034 10596 log.cc:824] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:01.048924 10595 tablet_bootstrap.cc:492] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803: No bootstrap required, opened a new log
I20260929 02:17:01.049207 10595 ts_tablet_manager.cc:1397] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 02:17:01.051162 10596 tablet_bootstrap.cc:492] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c: No bootstrap required, opened a new log
I20260929 02:17:01.051517 10596 ts_tablet_manager.cc:1397] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:17:01.052098 10599 tablet_bootstrap.cc:492] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1: Bootstrap starting.
I20260929 02:17:01.054935 10599 tablet_bootstrap.cc:654] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:01.055873 10599 log.cc:824] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:01.058475 10599 tablet_bootstrap.cc:492] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1: No bootstrap required, opened a new log
I20260929 02:17:01.058739 10599 ts_tablet_manager.cc:1397] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 02:17:01.060518 10595 raft_consensus.cc:348] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e53f686e1944c039f319d9cf068822c" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44565 } } peers { permanent_uuid: "8a279fc31af941bfa83575faddaa9803" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35617 } } peers { permanent_uuid: "6f9e618c69be4d86828c69d3316cd5b1" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 45033 } }
I20260929 02:17:01.060977 10595 raft_consensus.cc:374] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:01.061084 10595 raft_consensus.cc:729] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a279fc31af941bfa83575faddaa9803, State: Initialized, Role: FOLLOWER
I20260929 02:17:01.061709 10595 consensus_queue.cc:260] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e53f686e1944c039f319d9cf068822c" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44565 } } peers { permanent_uuid: "8a279fc31af941bfa83575faddaa9803" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35617 } } peers { permanent_uuid: "6f9e618c69be4d86828c69d3316cd5b1" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 45033 } }
I20260929 02:17:01.062032 10596 raft_consensus.cc:348] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e53f686e1944c039f319d9cf068822c" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44565 } } peers { permanent_uuid: "8a279fc31af941bfa83575faddaa9803" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35617 } } peers { permanent_uuid: "6f9e618c69be4d86828c69d3316cd5b1" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 45033 } }
I20260929 02:17:01.062565 10596 raft_consensus.cc:374] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:01.062737 10596 raft_consensus.cc:729] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e53f686e1944c039f319d9cf068822c, State: Initialized, Role: FOLLOWER
I20260929 02:17:01.062922 10577 heartbeater.cc:503] Master 127.24.216.190:36965 was elected leader, sending a full tablet report...
I20260929 02:17:01.063323 10596 consensus_queue.cc:260] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e53f686e1944c039f319d9cf068822c" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44565 } } peers { permanent_uuid: "8a279fc31af941bfa83575faddaa9803" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35617 } } peers { permanent_uuid: "6f9e618c69be4d86828c69d3316cd5b1" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 45033 } }
I20260929 02:17:01.066855 10599 raft_consensus.cc:348] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e53f686e1944c039f319d9cf068822c" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44565 } } peers { permanent_uuid: "8a279fc31af941bfa83575faddaa9803" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35617 } } peers { permanent_uuid: "6f9e618c69be4d86828c69d3316cd5b1" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 45033 } }
I20260929 02:17:01.067173 10599 raft_consensus.cc:374] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:01.067286 10599 raft_consensus.cc:729] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9e618c69be4d86828c69d3316cd5b1, State: Initialized, Role: FOLLOWER
I20260929 02:17:01.067945 10599 consensus_queue.cc:260] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e53f686e1944c039f319d9cf068822c" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44565 } } peers { permanent_uuid: "8a279fc31af941bfa83575faddaa9803" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35617 } } peers { permanent_uuid: "6f9e618c69be4d86828c69d3316cd5b1" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 45033 } }
I20260929 02:17:01.068327 10595 ts_tablet_manager.cc:1428] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803: Time spent starting tablet: real 0.019s user 0.012s sys 0.003s
I20260929 02:17:01.069989 10443 heartbeater.cc:503] Master 127.24.216.190:36965 was elected leader, sending a full tablet report...
I20260929 02:17:01.071101 10599 ts_tablet_manager.cc:1428] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20260929 02:17:01.071547 10603 raft_consensus.cc:482] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:17:01.071890 10596 ts_tablet_manager.cc:1428] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260929 02:17:01.071866 10603 raft_consensus.cc:504] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e53f686e1944c039f319d9cf068822c" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44565 } } peers { permanent_uuid: "8a279fc31af941bfa83575faddaa9803" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35617 } } peers { permanent_uuid: "6f9e618c69be4d86828c69d3316cd5b1" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 45033 } }
I20260929 02:17:01.075311 10603 leader_election.cc:288] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e53f686e1944c039f319d9cf068822c (127.24.216.130:44565), 8a279fc31af941bfa83575faddaa9803 (127.24.216.131:35617)
I20260929 02:17:01.081465 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:01.074792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:52691 (local address 127.24.216.131:35617)
0929 02:17:01.074988 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:01.074994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:01.075025 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:17:01.075240 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:01.075246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:01.075326 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:01.075400 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:01.076297 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.077302 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.079657 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.080631 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.080783 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:01.080855 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:01.081070 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 02:17:01.081211 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:17:01.081459 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:01.074266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:35617 (local address 127.24.216.129:52691)
0929 02:17:01.074903 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:01.074923 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:01.075053 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:01.075623 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:01.075628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:01.075654 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:01.076140 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:01.076153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.077418 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.077421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:01.079513 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:01.079528 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.080820 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.080825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:01.080932 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:01.080956 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:17:01.081029 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 02:17:01.081114 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":3,"thread_start_us":768,"threads_started":1}
I20260929 02:17:01.082281 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:01.074879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:36869 (local address 127.24.216.130:44565)
0929 02:17:01.075059 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:01.075066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:01.076347 (+ 1281us) server_negotiation.cc:407] Connection header received
0929 02:17:01.076469 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:01.076474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:01.076542 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:01.076648 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:01.077086 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.077881 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.080155 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.081294 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.081432 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:01.081515 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:01.081883 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 02:17:01.082007 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:17:01.082357 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:01.074333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:44565 (local address 127.24.216.129:36869)
0929 02:17:01.076251 (+ 1918us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:01.076270 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:01.076376 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:01.076659 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:01.076662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:01.076678 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:01.076957 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:01.076964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.078007 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.078012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:01.080025 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:01.080034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.081522 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.081528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:01.081704 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:01.081753 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 02:17:01.081857 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 02:17:01.081973 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1777,"thread_start_us":1573,"threads_started":1}
I20260929 02:17:01.082638 10532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a867141515e044f8b6b0f0364e583612" candidate_uuid: "6f9e618c69be4d86828c69d3316cd5b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a279fc31af941bfa83575faddaa9803" is_pre_election: true
I20260929 02:17:01.083079 10532 raft_consensus.cc:2454] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9e618c69be4d86828c69d3316cd5b1 in term 0.
I20260929 02:17:01.083582 10398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a867141515e044f8b6b0f0364e583612" candidate_uuid: "6f9e618c69be4d86828c69d3316cd5b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e53f686e1944c039f319d9cf068822c" is_pre_election: true
I20260929 02:17:01.083760 10198 leader_election.cc:302] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f9e618c69be4d86828c69d3316cd5b1, 8a279fc31af941bfa83575faddaa9803; no voters:
I20260929 02:17:01.084029 10398 raft_consensus.cc:2454] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9e618c69be4d86828c69d3316cd5b1 in term 0.
I20260929 02:17:01.084164 10603 raft_consensus.cc:2781] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:17:01.084349 10603 raft_consensus.cc:482] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:17:01.084461 10603 raft_consensus.cc:3037] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:01.086781 10603 raft_consensus.cc:504] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e53f686e1944c039f319d9cf068822c" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44565 } } peers { permanent_uuid: "8a279fc31af941bfa83575faddaa9803" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35617 } } peers { permanent_uuid: "6f9e618c69be4d86828c69d3316cd5b1" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 45033 } }
I20260929 02:17:01.087409 10603 leader_election.cc:288] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [CANDIDATE]: Term 1 election: Requested vote from peers 1e53f686e1944c039f319d9cf068822c (127.24.216.130:44565), 8a279fc31af941bfa83575faddaa9803 (127.24.216.131:35617)
I20260929 02:17:01.087711 10398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a867141515e044f8b6b0f0364e583612" candidate_uuid: "6f9e618c69be4d86828c69d3316cd5b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e53f686e1944c039f319d9cf068822c"
I20260929 02:17:01.087836 10532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a867141515e044f8b6b0f0364e583612" candidate_uuid: "6f9e618c69be4d86828c69d3316cd5b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a279fc31af941bfa83575faddaa9803"
I20260929 02:17:01.087895 10398 raft_consensus.cc:3037] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:01.088033 10532 raft_consensus.cc:3037] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:01.090119 10398 raft_consensus.cc:2454] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9e618c69be4d86828c69d3316cd5b1 in term 1.
I20260929 02:17:01.090139 10532 raft_consensus.cc:2454] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9e618c69be4d86828c69d3316cd5b1 in term 1.
I20260929 02:17:01.090596 10200 leader_election.cc:302] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e53f686e1944c039f319d9cf068822c, 6f9e618c69be4d86828c69d3316cd5b1, 8a279fc31af941bfa83575faddaa9803; no voters:
I20260929 02:17:01.090883 10603 raft_consensus.cc:2781] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:01.091313 10603 raft_consensus.cc:686] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [term 1 LEADER]: Becoming Leader. State: Replica: 6f9e618c69be4d86828c69d3316cd5b1, State: Running, Role: LEADER
I20260929 02:17:01.091756 10603 consensus_queue.cc:237] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e53f686e1944c039f319d9cf068822c" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44565 } } peers { permanent_uuid: "8a279fc31af941bfa83575faddaa9803" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35617 } } peers { permanent_uuid: "6f9e618c69be4d86828c69d3316cd5b1" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 45033 } }
I20260929 02:17:01.095746 10119 catalog_manager.cc:5495] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f9e618c69be4d86828c69d3316cd5b1 (127.24.216.129). New cstate: current_term: 1 leader_uuid: "6f9e618c69be4d86828c69d3316cd5b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e53f686e1944c039f319d9cf068822c" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a279fc31af941bfa83575faddaa9803" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f9e618c69be4d86828c69d3316cd5b1" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 45033 } health_report { overall_health: HEALTHY } } }
I20260929 02:17:01.131075 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:01.126128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34408 (local address 127.24.216.129:45033)
0929 02:17:01.126349 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:01.126354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:01.126371 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:17:01.126537 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:01.126541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:01.126658 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:01.126790 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:01.127093 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.127781 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.129040 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.129440 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.129528 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:01.129948 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:01.130522 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:01.130626 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:01.130725 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 02:17:01.130791 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 02:17:01.139065 10398 raft_consensus.cc:1261] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c [term 1 FOLLOWER]: Refusing update from remote peer 6f9e618c69be4d86828c69d3316cd5b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:17:01.139184 10532 raft_consensus.cc:1261] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803 [term 1 FOLLOWER]: Refusing update from remote peer 6f9e618c69be4d86828c69d3316cd5b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:17:01.140129 10606 consensus_queue.cc:1035] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a279fc31af941bfa83575faddaa9803" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:17:01.140498 10603 consensus_queue.cc:1035] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e53f686e1944c039f319d9cf068822c" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 44565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:17:01.151539 10609 mvcc.cc:205] Tried to move back new op lower bound from 7334495113775804416 to 7334495113595142144. Current Snapshot: MvccSnapshot[applied={T|T < 7334495113775804416}]
I20260929 02:17:01.155963 10611 mvcc.cc:205] Tried to move back new op lower bound from 7334495113775804416 to 7334495113595142144. Current Snapshot: MvccSnapshot[applied={T|T < 7334495113775804416}]
I20260929 02:17:01.156749 10610 mvcc.cc:205] Tried to move back new op lower bound from 7334495113775804416 to 7334495113595142144. Current Snapshot: MvccSnapshot[applied={T|T < 7334495113775804416 or (T in {7334495113775804416})}]
I20260929 02:17:01.169811 10119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:17:01.170086 10119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:17:01.173794 10119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d943decade394ee3a90438f7c8f7c560: Sending DeleteTablet for 3 replicas of tablet a867141515e044f8b6b0f0364e583612
I20260929 02:17:01.174965 10378 tablet_service.cc:1514] Processing DeleteTablet for tablet a867141515e044f8b6b0f0364e583612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:01 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:39684
I20260929 02:17:01.175025 10512 tablet_service.cc:1514] Processing DeleteTablet for tablet a867141515e044f8b6b0f0364e583612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:01 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:50362
I20260929 02:17:01.176301 10244 tablet_service.cc:1514] Processing DeleteTablet for tablet a867141515e044f8b6b0f0364e583612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:01 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:34404
I20260929 02:17:01.177049 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 10181
I20260929 02:17:01.177031 10619 tablet_replica.cc:330] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c: stopping tablet replica
I20260929 02:17:01.177546 10619 raft_consensus.cc:2229] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:17:01.178260 10619 raft_consensus.cc:2258] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:01.187057 10621 tablet_replica.cc:330] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1: stopping tablet replica
I20260929 02:17:01.187776 10621 raft_consensus.cc:2229] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:17:01.188175 10619 ts_tablet_manager.cc:1905] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:01.190506 10621 raft_consensus.cc:2258] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:01.192564 10621 ts_tablet_manager.cc:1905] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:01.196054 10620 tablet_replica.cc:330] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803: stopping tablet replica
I20260929 02:17:01.196475 10620 raft_consensus.cc:2229] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:17:01.196980 10620 raft_consensus.cc:2258] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:01.197023 10619 ts_tablet_manager.cc:1918] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:01.197248 10619 log.cc:1196] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/a867141515e044f8b6b0f0364e583612
I20260929 02:17:01.197853 10619 ts_tablet_manager.cc:1939] T a867141515e044f8b6b0f0364e583612 P 1e53f686e1944c039f319d9cf068822c: Deleting consensus metadata
I20260929 02:17:01.199280 10620 ts_tablet_manager.cc:1905] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:01.199832 10107 catalog_manager.cc:4841] TS 1e53f686e1944c039f319d9cf068822c (127.24.216.130:44565): tablet a867141515e044f8b6b0f0364e583612 (table test-table [id=3e4d354c36ba4414837317d84455aa28]) successfully deleted
Sep 29 02:17:01 dist-test-slave-9k0h krb5kdc[10075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:17:01.203539 10621 ts_tablet_manager.cc:1918] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:01.203805 10621 log.cc:1196] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/a867141515e044f8b6b0f0364e583612
I20260929 02:17:01.204476 10621 ts_tablet_manager.cc:1939] T a867141515e044f8b6b0f0364e583612 P 6f9e618c69be4d86828c69d3316cd5b1: Deleting consensus metadata
I20260929 02:17:01.206106 10620 ts_tablet_manager.cc:1918] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:01.206368 10620 log.cc:1196] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/a867141515e044f8b6b0f0364e583612
I20260929 02:17:01.206342 10104 catalog_manager.cc:4841] TS 6f9e618c69be4d86828c69d3316cd5b1 (127.24.216.129:45033): tablet a867141515e044f8b6b0f0364e583612 (table test-table [id=3e4d354c36ba4414837317d84455aa28]) successfully deleted
I20260929 02:17:01.206964 10620 ts_tablet_manager.cc:1939] T a867141515e044f8b6b0f0364e583612 P 8a279fc31af941bfa83575faddaa9803: Deleting consensus metadata
I20260929 02:17:01.208676 10105 catalog_manager.cc:4841] TS 8a279fc31af941bfa83575faddaa9803 (127.24.216.131:35617): tablet a867141515e044f8b6b0f0364e583612 (table test-table [id=3e4d354c36ba4414837317d84455aa28]) successfully deleted
I20260929 02:17:01.209506 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:01.180008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.24.216.129:45033)
0929 02:17:01.180218 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:01.180227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:01.186586 (+ 6359us) server_negotiation.cc:407] Connection header received
0929 02:17:01.188006 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:01.188013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:01.188102 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:01.188182 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:01.190893 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.192010 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.194000 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.194712 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.194808 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:01.201775 (+ 6967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:01.201788 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:01.201805 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:01.201853 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:01.206014 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:01.206922 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:01.206930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:01.206934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:01.207127 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:01.207511 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:01.207516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:01.207519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:01.207982 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:01.208277 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:01.209057 (+ 780us) server_negotiation.cc:299] Negotiation successful
0929 02:17:01.209251 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:17:01.210210 10304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:01.307657 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 10181
I20260929 02:17:01.326532 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 10315
Sep 29 02:17:01 dist-test-slave-9k0h krb5kdc[10075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:17:01.341317 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:01.327333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.24.216.130:44565)
0929 02:17:01.327524 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:01.327528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:01.327548 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:17:01.327647 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:01.327651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:01.327709 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:01.327791 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:01.329107 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.329839 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.331188 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.331579 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.331677 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:01.335769 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:01.335782 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:01.335797 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:01.335834 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:01.338901 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:01.339510 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:01.339517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:01.339519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:01.339637 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:01.339947 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:01.339950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:01.339952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:01.340313 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:01.340527 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:01.340783 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 02:17:01.340942 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:17:01.341951 10438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:01.432703 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 10315
I20260929 02:17:01.451535 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 10449
Sep 29 02:17:01 dist-test-slave-9k0h krb5kdc[10075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:17:01.466251 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:01.452309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50368 (local address 127.24.216.131:35617)
0929 02:17:01.452501 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:01.452508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:01.452529 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:17:01.452703 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:01.452709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:01.452797 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:01.452904 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:01.454315 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.455048 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.456332 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:01.456746 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:01.456883 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:01.461170 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:01.461185 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:01.461205 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:01.461237 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:01.464008 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:01.464578 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:01.464584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:01.464587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:01.464716 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:01.465013 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:01.465015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:01.465017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:01.465322 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:01.465541 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:01.465794 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 02:17:01.465950 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:17:01.466889 10572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:01.557196 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 10449
I20260929 02:17:01.574582 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 10088
I20260929 02:17:01.575713 10150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:01.685354 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 10088
2026-09-29T02:17:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4238 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T02:17:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:01Z Disabled control of system clock
I20260929 02:17:01.775745 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:35835
--webserver_interface=127.24.216.190
--webserver_port=0
--builtin_ntp_servers=127.24.216.148:39017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:35835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:17:02.105348 10630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:02.105641 10630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:02.105726 10630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:02.113044 10630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:02.113165 10630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:02.113209 10630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:02.113260 10630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:02.113302 10630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:02.122859 10630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:39017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:35835
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:35835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:02.124181 10630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:02.125909 10630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:02.138175 10636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:02.138252 10635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:02.138356 10630 server_base.cc:1034] running on GCE node
W20260929 02:17:02.138216 10638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:02.139467 10630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:02.140815 10630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:02.142045 10630 hybrid_clock.cc:648] HybridClock initialized: now 1790648222142000 us; error 37 us; skew 500 ppm
I20260929 02:17:02.142405 10630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:17:02.144611 10630 webserver.cc:460] Webserver started at http://127.24.216.190:39537/ using document root <none> and password file <none>
I20260929 02:17:02.145273 10630 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:02.145397 10630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:02.145716 10630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:02.148622 10630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "0b062215cf3947a9a3e24769bfe5be30"
format_stamp: "Formatted at 2026-09-29 02:17:02 on dist-test-slave-9k0h"
server_key: "4bbd817476e901fe714fdc1610238439"
server_key_iv: "da7d70d18ed4c2aee6a37ffd75c98867"
server_key_version: "encryptionkey@0"
I20260929 02:17:02.149415 10630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "0b062215cf3947a9a3e24769bfe5be30"
format_stamp: "Formatted at 2026-09-29 02:17:02 on dist-test-slave-9k0h"
server_key: "4bbd817476e901fe714fdc1610238439"
server_key_iv: "da7d70d18ed4c2aee6a37ffd75c98867"
server_key_version: "encryptionkey@0"
I20260929 02:17:02.154349 10630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:17:02.157666 10644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:02.158937 10630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:17:02.159111 10630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "0b062215cf3947a9a3e24769bfe5be30"
format_stamp: "Formatted at 2026-09-29 02:17:02 on dist-test-slave-9k0h"
server_key: "4bbd817476e901fe714fdc1610238439"
server_key_iv: "da7d70d18ed4c2aee6a37ffd75c98867"
server_key_version: "encryptionkey@0"
I20260929 02:17:02.159324 10630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:02.176329 10630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:02.177142 10630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:02.177419 10630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:02.195353 10630 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:35835
I20260929 02:17:02.195359 10695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:35835 every 8 connection(s)
I20260929 02:17:02.196682 10630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:02.199198 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 10630
I20260929 02:17:02.199329 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:02.199780 25442 external_mini_cluster.cc:1428] Setting key 6197ab5e5cc32bd45b65f63c3a09ae13
I20260929 02:17:02.200562 10696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:02.215065 10696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30: Bootstrap starting.
I20260929 02:17:02.215062 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:02.201697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36970 (local address 127.24.216.190:35835)
0929 02:17:02.202739 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:02.202754 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:02.202803 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:17:02.205126 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:02.205156 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:02.205740 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:02.206197 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:02.206727 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:02.208724 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:02.210057 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:02.211130 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:02.211321 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:02.212337 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:02.212352 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:02.212374 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:02.213091 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:02.213184 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:02.213209 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:02.213308 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:02.213498 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 02:17:02.213811 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":319,"threads_started":1}
I20260929 02:17:02.218329 10696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:02.219272 10696 log.cc:824] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:02.221851 10696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30: No bootstrap required, opened a new log
I20260929 02:17:02.229242 10696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b062215cf3947a9a3e24769bfe5be30" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35835 } }
I20260929 02:17:02.229557 10696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:02.229629 10696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b062215cf3947a9a3e24769bfe5be30, State: Initialized, Role: FOLLOWER
I20260929 02:17:02.230082 10696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b062215cf3947a9a3e24769bfe5be30" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35835 } }
I20260929 02:17:02.230347 10696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:02.230448 10696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:02.230563 10696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:02.232620 10696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b062215cf3947a9a3e24769bfe5be30" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35835 } }
I20260929 02:17:02.233222 10696 leader_election.cc:302] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b062215cf3947a9a3e24769bfe5be30; no voters:
I20260929 02:17:02.233783 10696 leader_election.cc:288] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:02.233918 10701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:02.234644 10701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [term 1 LEADER]: Becoming Leader. State: Replica: 0b062215cf3947a9a3e24769bfe5be30, State: Running, Role: LEADER
I20260929 02:17:02.235144 10701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b062215cf3947a9a3e24769bfe5be30" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35835 } }
I20260929 02:17:02.235800 10696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:02.237735 10702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b062215cf3947a9a3e24769bfe5be30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b062215cf3947a9a3e24769bfe5be30" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35835 } } }
I20260929 02:17:02.237758 10703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b062215cf3947a9a3e24769bfe5be30. Latest consensus state: current_term: 1 leader_uuid: "0b062215cf3947a9a3e24769bfe5be30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b062215cf3947a9a3e24769bfe5be30" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35835 } } }
I20260929 02:17:02.238204 10702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:02.238277 10703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:02.239900 10706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:02.244026 10706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:02.252233 10706 catalog_manager.cc:1347] Generated new cluster ID: 1ee2c88fca1447a69bd7fd87a02da11f
I20260929 02:17:02.252338 10706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:02.277827 10706 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:02.279168 10706 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:02.284216 10706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30: Generated new TSK 0
I20260929 02:17:02.285199 10706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:02.290388 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35835
--builtin_ntp_servers=127.24.216.148:39017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:17:02.617218 10720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:02.617476 10720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:02.617630 10720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:02.624799 10720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:02.624891 10720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:02.625084 10720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:17:02.633989 10720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:39017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:02.635278 10720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:02.637205 10720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:02.649837 10725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:02.649798 10728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:02.649880 10726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:02.655817 10720 server_base.cc:1034] running on GCE node
I20260929 02:17:02.656419 10720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:02.657354 10720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:02.658545 10720 hybrid_clock.cc:648] HybridClock initialized: now 1790648222658504 us; error 29 us; skew 500 ppm
I20260929 02:17:02.658854 10720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:17:02.660997 10720 webserver.cc:460] Webserver started at http://127.24.216.129:40077/ using document root <none> and password file <none>
I20260929 02:17:02.661676 10720 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:02.661788 10720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:02.662137 10720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:02.665112 10720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "d60083dd50a34ce184e7763a5a4668c0"
format_stamp: "Formatted at 2026-09-29 02:17:02 on dist-test-slave-9k0h"
server_key: "f848d49dcd56a847de9a116b6dd5cb71"
server_key_iv: "b55242abc6ebe2e5c42a1e720ae57236"
server_key_version: "encryptionkey@0"
I20260929 02:17:02.665855 10720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "d60083dd50a34ce184e7763a5a4668c0"
format_stamp: "Formatted at 2026-09-29 02:17:02 on dist-test-slave-9k0h"
server_key: "f848d49dcd56a847de9a116b6dd5cb71"
server_key_iv: "b55242abc6ebe2e5c42a1e720ae57236"
server_key_version: "encryptionkey@0"
I20260929 02:17:02.670632 10720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:17:02.673889 10734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:02.675099 10720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:17:02.675271 10720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "d60083dd50a34ce184e7763a5a4668c0"
format_stamp: "Formatted at 2026-09-29 02:17:02 on dist-test-slave-9k0h"
server_key: "f848d49dcd56a847de9a116b6dd5cb71"
server_key_iv: "b55242abc6ebe2e5c42a1e720ae57236"
server_key_version: "encryptionkey@0"
I20260929 02:17:02.675529 10720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:02.690976 10720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:02.691833 10720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:02.692168 10720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:02.693354 10720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:02.695093 10720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:02.695212 10720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:02.695325 10720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:02.695402 10720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:02.734294 10720 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:42021
I20260929 02:17:02.734355 10846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:42021 every 8 connection(s)
I20260929 02:17:02.736147 10720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:17:02.746294 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 10720
I20260929 02:17:02.746481 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:17:02.746917 25442 external_mini_cluster.cc:1428] Setting key d262feb7e77c826df4b03b4147ffe15b
I20260929 02:17:02.749532 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:02.739123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:45017 (local address 127.24.216.190:35835)
0929 02:17:02.739588 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:02.739594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:02.741034 (+ 1440us) server_negotiation.cc:407] Connection header received
0929 02:17:02.742936 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:02.742942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:02.743046 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:02.743172 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:02.744177 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:02.744908 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:02.746607 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:02.747096 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:02.747234 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:02.748815 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:02.748821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:02.748825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:02.748846 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:02.748894 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:02.748911 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:02.748996 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:02.749238 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 02:17:02.749310 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":220,"threads_started":1}
I20260929 02:17:02.750015 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35835
--builtin_ntp_servers=127.24.216.148:39017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:17:02.751101 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:02.739723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:35835 (local address 127.24.216.129:45017)
0929 02:17:02.740844 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:02.740898 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:02.742667 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:02.743396 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:02.743404 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:02.743484 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:02.744016 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:02.744032 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:02.745080 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:02.745084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:02.746395 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:02.746425 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:02.747336 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:02.747345 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:02.747690 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:02.748452 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:17:02.748454 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:02.748492 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:17:02.748606 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:02.749019 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:02.749030 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:02.749037 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:17:02.749227 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 02:17:02.749619 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":315,"threads_started":1}
I20260929 02:17:02.752569 10847 heartbeater.cc:347] Connected to a master server at 127.24.216.190:35835
I20260929 02:17:02.752878 10847 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:02.753605 10847 heartbeater.cc:511] Master 127.24.216.190:35835 requested a full tablet report, sending...
I20260929 02:17:02.755548 10661 ts_manager.cc:194] Registered new tserver with Master: d60083dd50a34ce184e7763a5a4668c0 (127.24.216.129:42021)
I20260929 02:17:02.756963 10661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.216.129:45017
W20260929 02:17:03.086911 10851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:03.087184 10851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:03.087322 10851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:03.094337 10851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:03.094434 10851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:03.094612 10851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:17:03.103415 10851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:39017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:03.104652 10851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:03.106472 10851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:03.119444 10856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:03.119570 10857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:03.119766 10859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:03.120411 10851 server_base.cc:1034] running on GCE node
I20260929 02:17:03.121009 10851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:03.121940 10851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:03.123163 10851 hybrid_clock.cc:648] HybridClock initialized: now 1790648223123093 us; error 66 us; skew 500 ppm
I20260929 02:17:03.123485 10851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:17:03.125718 10851 webserver.cc:460] Webserver started at http://127.24.216.130:34575/ using document root <none> and password file <none>
I20260929 02:17:03.126474 10851 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:03.126596 10851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:03.126907 10851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:03.129704 10851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "5de1ac7dff4b47cd9d446f6309c309c4"
format_stamp: "Formatted at 2026-09-29 02:17:03 on dist-test-slave-9k0h"
server_key: "10545894a179186e0e0231402fabf83a"
server_key_iv: "c7220be346f24bf5e64abb3d65de1f92"
server_key_version: "encryptionkey@0"
I20260929 02:17:03.130553 10851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "5de1ac7dff4b47cd9d446f6309c309c4"
format_stamp: "Formatted at 2026-09-29 02:17:03 on dist-test-slave-9k0h"
server_key: "10545894a179186e0e0231402fabf83a"
server_key_iv: "c7220be346f24bf5e64abb3d65de1f92"
server_key_version: "encryptionkey@0"
I20260929 02:17:03.135577 10851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:17:03.139101 10865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:03.140424 10851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:17:03.140588 10851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "5de1ac7dff4b47cd9d446f6309c309c4"
format_stamp: "Formatted at 2026-09-29 02:17:03 on dist-test-slave-9k0h"
server_key: "10545894a179186e0e0231402fabf83a"
server_key_iv: "c7220be346f24bf5e64abb3d65de1f92"
server_key_version: "encryptionkey@0"
I20260929 02:17:03.140806 10851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:03.174266 10851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:03.175134 10851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:03.175426 10851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:03.176417 10851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:03.178048 10851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:03.178133 10851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:03.178292 10851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:03.178385 10851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:03.211786 10851 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:33251
I20260929 02:17:03.211923 10977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:33251 every 8 connection(s)
I20260929 02:17:03.213053 10851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:17:03.218266 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 10851
I20260929 02:17:03.218448 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:17:03.219080 25442 external_mini_cluster.cc:1428] Setting key 3a7e72be8b53324424281b6a0581d210
I20260929 02:17:03.223019 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35835
--builtin_ntp_servers=127.24.216.148:39017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:17:03.227732 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.216023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:36933 (local address 127.24.216.190:35835)
0929 02:17:03.216266 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:03.216274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:03.218083 (+ 1809us) server_negotiation.cc:407] Connection header received
0929 02:17:03.220783 (+ 2700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:03.220790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:03.220873 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:03.220978 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:03.222442 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.223170 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.224709 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.225453 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.225608 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.226858 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:03.226867 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:03.226874 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:03.226898 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:03.226940 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:03.226964 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:03.227051 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:03.227251 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 02:17:03.227359 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:17:03.228503 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.216570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:35835 (local address 127.24.216.130:36933)
0929 02:17:03.217908 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:03.217970 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:03.220394 (+ 2424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:03.221309 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:03.221324 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:03.221459 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:03.222270 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:03.222292 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.223363 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.223366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:03.224566 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:03.224577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.225641 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.225645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:03.225838 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.226565 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:17:03.226567 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:03.226603 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:17:03.226677 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:03.227061 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:03.227068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:03.227073 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:17:03.227206 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 02:17:03.227470 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":350,"threads_started":1}
I20260929 02:17:03.229832 10978 heartbeater.cc:347] Connected to a master server at 127.24.216.190:35835
I20260929 02:17:03.230131 10978 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:03.231029 10978 heartbeater.cc:511] Master 127.24.216.190:35835 requested a full tablet report, sending...
I20260929 02:17:03.232522 10661 ts_manager.cc:194] Registered new tserver with Master: 5de1ac7dff4b47cd9d446f6309c309c4 (127.24.216.130:33251)
I20260929 02:17:03.233376 10661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.216.130:36933
W20260929 02:17:03.551982 10981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:03.552249 10981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:03.552398 10981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:03.559530 10981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:03.559628 10981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:03.559799 10981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:17:03.568691 10981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:39017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:03.569926 10981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:03.572162 10981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:03.585191 10986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:03.585168 10987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:03.585255 10989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:03.585949 10981 server_base.cc:1034] running on GCE node
I20260929 02:17:03.586563 10981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:03.587487 10981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:03.588707 10981 hybrid_clock.cc:648] HybridClock initialized: now 1790648223588678 us; error 38 us; skew 500 ppm
I20260929 02:17:03.589003 10981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:17:03.591099 10981 webserver.cc:460] Webserver started at http://127.24.216.131:33573/ using document root <none> and password file <none>
I20260929 02:17:03.591751 10981 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:03.591876 10981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:03.592182 10981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:03.594960 10981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "b3c5356d965f471cab09583564cb62a1"
format_stamp: "Formatted at 2026-09-29 02:17:03 on dist-test-slave-9k0h"
server_key: "6f4611d03d3b318c24d76b5dc73f57cf"
server_key_iv: "8ab08f3305503caa72edc76dc2e75130"
server_key_version: "encryptionkey@0"
I20260929 02:17:03.595724 10981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "b3c5356d965f471cab09583564cb62a1"
format_stamp: "Formatted at 2026-09-29 02:17:03 on dist-test-slave-9k0h"
server_key: "6f4611d03d3b318c24d76b5dc73f57cf"
server_key_iv: "8ab08f3305503caa72edc76dc2e75130"
server_key_version: "encryptionkey@0"
I20260929 02:17:03.600443 10981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:17:03.603637 10995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:03.604810 10981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 02:17:03.604969 10981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "b3c5356d965f471cab09583564cb62a1"
format_stamp: "Formatted at 2026-09-29 02:17:03 on dist-test-slave-9k0h"
server_key: "6f4611d03d3b318c24d76b5dc73f57cf"
server_key_iv: "8ab08f3305503caa72edc76dc2e75130"
server_key_version: "encryptionkey@0"
I20260929 02:17:03.605175 10981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:03.630276 10981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:03.631068 10981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:03.631376 10981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:03.632309 10981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:03.633706 10981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:03.633828 10981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:03.633946 10981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:03.634032 10981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:03.666621 10981 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:45913
I20260929 02:17:03.666705 11107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:45913 every 8 connection(s)
I20260929 02:17:03.667897 10981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:17:03.670011 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 10981
I20260929 02:17:03.670221 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:17:03.670804 25442 external_mini_cluster.cc:1428] Setting key 456c3bfa17111ba60efd4177ed157de5
I20260929 02:17:03.679754 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.670288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:42837 (local address 127.24.216.190:35835)
0929 02:17:03.670505 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:03.670514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:03.671958 (+ 1444us) server_negotiation.cc:407] Connection header received
0929 02:17:03.674085 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:03.674089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:03.674194 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:03.674278 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:03.675246 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.675952 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.677383 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.677775 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.677862 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.679135 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:03.679140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:03.679143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:03.679158 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:03.679198 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:03.679209 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:03.679276 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:03.679453 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 02:17:03.679519 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:17:03.680751 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.670823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:35835 (local address 127.24.216.131:42837)
0929 02:17:03.671815 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:03.671861 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:03.673857 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:03.674441 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:03.674449 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:03.674534 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:03.675113 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:03.675130 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.676066 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.676069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:03.677269 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:03.677280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.677962 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.677966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:03.678217 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.678863 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:17:03.678865 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:03.678901 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:17:03.678973 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:03.679287 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:03.679294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:03.679299 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:17:03.679430 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 02:17:03.679687 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":283,"threads_started":1}
I20260929 02:17:03.681973 11108 heartbeater.cc:347] Connected to a master server at 127.24.216.190:35835
I20260929 02:17:03.682312 11108 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:03.682988 11108 heartbeater.cc:511] Master 127.24.216.190:35835 requested a full tablet report, sending...
I20260929 02:17:03.684491 10661 ts_manager.cc:194] Registered new tserver with Master: b3c5356d965f471cab09583564cb62a1 (127.24.216.131:45913)
I20260929 02:17:03.685490 10661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.216.131:42837
I20260929 02:17:03.686039 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:17:03.695598 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.690026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.24.216.190:35835)
0929 02:17:03.690276 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:03.690284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:03.690803 (+ 519us) server_negotiation.cc:407] Connection header received
0929 02:17:03.690926 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:03.690930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:03.691009 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:03.691098 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:03.691520 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.692205 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.693473 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.693878 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.693995 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.695023 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:03.695028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:03.695032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:03.695049 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:03.695080 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:03.695092 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:03.695170 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:03.695288 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 02:17:03.695358 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:17:03.699100 10661 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:17:03.705672 10661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:17:03.731818 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.716717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:33251 (local address 127.0.0.1:42414)
0929 02:17:03.717676 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:03.717750 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:03.717958 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:03.724353 (+ 6395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:03.724360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:03.724386 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:03.724794 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:03.724812 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.726744 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.726750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:03.728682 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:03.728705 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.730865 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.730871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:03.731063 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.731115 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 02:17:03.731256 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 02:17:03.731422 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":3,"thread_start_us":7099,"threads_started":1}
I20260929 02:17:03.731979 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.717190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.24.216.130:33251)
0929 02:17:03.722638 (+ 5448us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:03.722651 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:03.722692 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:17:03.722813 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:03.722824 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:03.723135 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:03.723373 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:03.724997 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.726599 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.728846 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.730348 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.730512 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.730729 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:03.731450 (+ 721us) server_negotiation.cc:299] Negotiation successful
0929 02:17:03.731640 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5257,"thread_start_us":5137,"threads_started":1}
I20260929 02:17:03.733217 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.719178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42112 (local address 127.24.216.129:42021)
0929 02:17:03.725133 (+ 5955us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:03.725140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:03.725165 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:17:03.725262 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:03.725272 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:03.725473 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:03.725666 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:03.726308 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.727775 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.730133 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.731564 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.731707 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.732483 (+ 776us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:03.732702 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 02:17:03.732878 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5789,"thread_start_us":5660,"threads_started":1}
I20260929 02:17:03.733668 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.718601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:42021 (local address 127.0.0.1:42112)
0929 02:17:03.724052 (+ 5451us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:03.724069 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:03.724174 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:03.725680 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:03.725687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:03.725731 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:03.726101 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:03.726112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.727925 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.727932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:03.729912 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:03.729924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.732331 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.732338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:03.732517 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.732552 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:17:03.732643 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 02:17:03.732763 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5336,"mutex_wait_us":1,"thread_start_us":767,"threads_started":1}
I20260929 02:17:03.734349 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.716925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:45913 (local address 127.0.0.1:57372)
0929 02:17:03.719181 (+ 2256us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:03.719209 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:03.719338 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:03.727108 (+ 7770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:03.727117 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:03.727145 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:03.727621 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:03.727634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.729543 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.729550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:03.731606 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:03.731619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.733626 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.733635 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:03.733831 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.733860 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:17:03.733939 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 02:17:03.734064 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2091,"thread_start_us":259,"threads_started":1}
I20260929 02:17:03.735252 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.717147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57372 (local address 127.24.216.131:45913)
0929 02:17:03.726324 (+ 9177us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:03.726332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:03.726362 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:17:03.726443 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:03.726450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:03.726645 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:03.726813 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:03.727828 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.729405 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.731843 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.733228 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.733497 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.734741 (+ 1244us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:03.734804 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 02:17:03.734948 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9018,"thread_start_us":8840,"threads_started":1}
I20260929 02:17:03.736080 10913 tablet_service.cc:1467] Processing CreateTablet for tablet 0103c7077f144d6fba478665f9855a85 (DEFAULT_TABLE table=test-table [id=530dc304f4e347efbf6e4a8598b8fe9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:03.737336 10913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0103c7077f144d6fba478665f9855a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:03.738768 10782 tablet_service.cc:1467] Processing CreateTablet for tablet 0103c7077f144d6fba478665f9855a85 (DEFAULT_TABLE table=test-table [id=530dc304f4e347efbf6e4a8598b8fe9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:03.740165 10782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0103c7077f144d6fba478665f9855a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:03.745715 11043 tablet_service.cc:1467] Processing CreateTablet for tablet 0103c7077f144d6fba478665f9855a85 (DEFAULT_TABLE table=test-table [id=530dc304f4e347efbf6e4a8598b8fe9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:03.747094 11043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0103c7077f144d6fba478665f9855a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:03.749840 11126 tablet_bootstrap.cc:492] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4: Bootstrap starting.
I20260929 02:17:03.751945 11127 tablet_bootstrap.cc:492] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0: Bootstrap starting.
I20260929 02:17:03.753720 11126 tablet_bootstrap.cc:654] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:03.754761 11126 log.cc:824] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:03.755241 11127 tablet_bootstrap.cc:654] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:03.756388 11127 log.cc:824] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:03.757575 11126 tablet_bootstrap.cc:492] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4: No bootstrap required, opened a new log
I20260929 02:17:03.757827 11126 ts_tablet_manager.cc:1397] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 02:17:03.759186 11127 tablet_bootstrap.cc:492] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0: No bootstrap required, opened a new log
I20260929 02:17:03.759440 11127 ts_tablet_manager.cc:1397] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 02:17:03.759780 10847 heartbeater.cc:503] Master 127.24.216.190:35835 was elected leader, sending a full tablet report...
I20260929 02:17:03.761494 11130 tablet_bootstrap.cc:492] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1: Bootstrap starting.
I20260929 02:17:03.765003 11130 tablet_bootstrap.cc:654] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:03.766218 11130 log.cc:824] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:03.767593 11126 raft_consensus.cc:348] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de1ac7dff4b47cd9d446f6309c309c4" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33251 } } peers { permanent_uuid: "b3c5356d965f471cab09583564cb62a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 45913 } } peers { permanent_uuid: "d60083dd50a34ce184e7763a5a4668c0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42021 } }
I20260929 02:17:03.768038 11126 raft_consensus.cc:374] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:03.768141 11126 raft_consensus.cc:729] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de1ac7dff4b47cd9d446f6309c309c4, State: Initialized, Role: FOLLOWER
I20260929 02:17:03.768776 11126 consensus_queue.cc:260] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de1ac7dff4b47cd9d446f6309c309c4" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33251 } } peers { permanent_uuid: "b3c5356d965f471cab09583564cb62a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 45913 } } peers { permanent_uuid: "d60083dd50a34ce184e7763a5a4668c0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42021 } }
I20260929 02:17:03.769301 11130 tablet_bootstrap.cc:492] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1: No bootstrap required, opened a new log
I20260929 02:17:03.769573 11130 ts_tablet_manager.cc:1397] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 02:17:03.769969 10978 heartbeater.cc:503] Master 127.24.216.190:35835 was elected leader, sending a full tablet report...
I20260929 02:17:03.770573 11126 ts_tablet_manager.cc:1428] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260929 02:17:03.770812 11127 raft_consensus.cc:348] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de1ac7dff4b47cd9d446f6309c309c4" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33251 } } peers { permanent_uuid: "b3c5356d965f471cab09583564cb62a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 45913 } } peers { permanent_uuid: "d60083dd50a34ce184e7763a5a4668c0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42021 } }
I20260929 02:17:03.771235 11127 raft_consensus.cc:374] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:03.771337 11127 raft_consensus.cc:729] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60083dd50a34ce184e7763a5a4668c0, State: Initialized, Role: FOLLOWER
I20260929 02:17:03.771919 11127 consensus_queue.cc:260] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de1ac7dff4b47cd9d446f6309c309c4" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33251 } } peers { permanent_uuid: "b3c5356d965f471cab09583564cb62a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 45913 } } peers { permanent_uuid: "d60083dd50a34ce184e7763a5a4668c0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42021 } }
I20260929 02:17:03.773517 11127 ts_tablet_manager.cc:1428] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260929 02:17:03.778349 11130 raft_consensus.cc:348] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de1ac7dff4b47cd9d446f6309c309c4" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33251 } } peers { permanent_uuid: "b3c5356d965f471cab09583564cb62a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 45913 } } peers { permanent_uuid: "d60083dd50a34ce184e7763a5a4668c0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42021 } }
I20260929 02:17:03.778764 11130 raft_consensus.cc:374] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:03.778882 11130 raft_consensus.cc:729] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c5356d965f471cab09583564cb62a1, State: Initialized, Role: FOLLOWER
I20260929 02:17:03.779450 11130 consensus_queue.cc:260] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de1ac7dff4b47cd9d446f6309c309c4" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33251 } } peers { permanent_uuid: "b3c5356d965f471cab09583564cb62a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 45913 } } peers { permanent_uuid: "d60083dd50a34ce184e7763a5a4668c0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42021 } }
I20260929 02:17:03.780367 11108 heartbeater.cc:503] Master 127.24.216.190:35835 was elected leader, sending a full tablet report...
I20260929 02:17:03.780838 11130 ts_tablet_manager.cc:1428] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 02:17:03.864447 11133 raft_consensus.cc:482] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:17:03.864746 11133 raft_consensus.cc:504] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de1ac7dff4b47cd9d446f6309c309c4" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33251 } } peers { permanent_uuid: "b3c5356d965f471cab09583564cb62a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 45913 } } peers { permanent_uuid: "d60083dd50a34ce184e7763a5a4668c0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42021 } }
I20260929 02:17:03.866415 11133 leader_election.cc:288] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5de1ac7dff4b47cd9d446f6309c309c4 (127.24.216.130:33251), b3c5356d965f471cab09583564cb62a1 (127.24.216.131:45913)
I20260929 02:17:03.873006 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.867062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:33283 (local address 127.24.216.131:45913)
0929 02:17:03.867245 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:03.867253 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:03.867606 (+ 353us) server_negotiation.cc:407] Connection header received
0929 02:17:03.867782 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:03.867788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:03.867871 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:03.867952 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:03.868565 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.869304 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.871287 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.872059 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.872235 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.872319 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:03.872622 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 02:17:03.872764 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 02:17:03.873023 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.866857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:57305 (local address 127.24.216.130:33251)
0929 02:17:03.867293 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:03.867299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:03.867745 (+ 446us) server_negotiation.cc:407] Connection header received
0929 02:17:03.867912 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:03.867916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:03.868019 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:03.868113 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:03.868726 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.869434 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.871349 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.872126 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.872274 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.872345 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:03.872622 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 02:17:03.872755 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285}
I20260929 02:17:03.873152 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.866841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:33251 (local address 127.24.216.129:57305)
0929 02:17:03.867623 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:03.867640 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:03.867788 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:03.868137 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:03.868141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:03.868167 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:03.868552 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:03.868562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.869563 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.869567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:03.871099 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:03.871110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.872315 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.872321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:03.872469 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.872501 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:17:03.872595 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 02:17:03.872765 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":62,"thread_start_us":489,"threads_started":1}
I20260929 02:17:03.873310 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.866686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:45913 (local address 127.24.216.129:33283)
0929 02:17:03.867474 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:03.867494 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:03.867644 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:03.867998 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:03.868004 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:03.868027 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:03.868428 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:03.868438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.869437 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.869442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:03.871096 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:03.871107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.872250 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.872256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:03.872419 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.872498 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 02:17:03.872592 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 02:17:03.872748 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":115,"thread_start_us":557,"threads_started":1}
I20260929 02:17:03.874290 11063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0103c7077f144d6fba478665f9855a85" candidate_uuid: "d60083dd50a34ce184e7763a5a4668c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c5356d965f471cab09583564cb62a1" is_pre_election: true
I20260929 02:17:03.874594 10933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0103c7077f144d6fba478665f9855a85" candidate_uuid: "d60083dd50a34ce184e7763a5a4668c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de1ac7dff4b47cd9d446f6309c309c4" is_pre_election: true
I20260929 02:17:03.874789 11063 raft_consensus.cc:2454] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d60083dd50a34ce184e7763a5a4668c0 in term 0.
I20260929 02:17:03.875087 10933 raft_consensus.cc:2454] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d60083dd50a34ce184e7763a5a4668c0 in term 0.
I20260929 02:17:03.875514 10735 leader_election.cc:302] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3c5356d965f471cab09583564cb62a1, d60083dd50a34ce184e7763a5a4668c0; no voters:
I20260929 02:17:03.875893 11133 raft_consensus.cc:2781] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:17:03.876051 11133 raft_consensus.cc:482] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:17:03.876204 11133 raft_consensus.cc:3037] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:03.878670 11133 raft_consensus.cc:504] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de1ac7dff4b47cd9d446f6309c309c4" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33251 } } peers { permanent_uuid: "b3c5356d965f471cab09583564cb62a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 45913 } } peers { permanent_uuid: "d60083dd50a34ce184e7763a5a4668c0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42021 } }
I20260929 02:17:03.879184 11133 leader_election.cc:288] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [CANDIDATE]: Term 1 election: Requested vote from peers 5de1ac7dff4b47cd9d446f6309c309c4 (127.24.216.130:33251), b3c5356d965f471cab09583564cb62a1 (127.24.216.131:45913)
I20260929 02:17:03.879623 10933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0103c7077f144d6fba478665f9855a85" candidate_uuid: "d60083dd50a34ce184e7763a5a4668c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5de1ac7dff4b47cd9d446f6309c309c4"
I20260929 02:17:03.879706 11063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0103c7077f144d6fba478665f9855a85" candidate_uuid: "d60083dd50a34ce184e7763a5a4668c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c5356d965f471cab09583564cb62a1"
I20260929 02:17:03.879830 10933 raft_consensus.cc:3037] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:03.879935 11063 raft_consensus.cc:3037] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:03.882308 11063 raft_consensus.cc:2454] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d60083dd50a34ce184e7763a5a4668c0 in term 1.
I20260929 02:17:03.882321 10933 raft_consensus.cc:2454] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d60083dd50a34ce184e7763a5a4668c0 in term 1.
I20260929 02:17:03.882689 10735 leader_election.cc:302] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3c5356d965f471cab09583564cb62a1, d60083dd50a34ce184e7763a5a4668c0; no voters:
I20260929 02:17:03.882977 11133 raft_consensus.cc:2781] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:03.883399 11133 raft_consensus.cc:686] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [term 1 LEADER]: Becoming Leader. State: Replica: d60083dd50a34ce184e7763a5a4668c0, State: Running, Role: LEADER
I20260929 02:17:03.883826 11133 consensus_queue.cc:237] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de1ac7dff4b47cd9d446f6309c309c4" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33251 } } peers { permanent_uuid: "b3c5356d965f471cab09583564cb62a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 45913 } } peers { permanent_uuid: "d60083dd50a34ce184e7763a5a4668c0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42021 } }
I20260929 02:17:03.888342 10659 catalog_manager.cc:5495] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d60083dd50a34ce184e7763a5a4668c0 (127.24.216.129). New cstate: current_term: 1 leader_uuid: "d60083dd50a34ce184e7763a5a4668c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5de1ac7dff4b47cd9d446f6309c309c4" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3c5356d965f471cab09583564cb62a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 45913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d60083dd50a34ce184e7763a5a4668c0" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42021 } health_report { overall_health: HEALTHY } } }
W20260929 02:17:03.919600 11109 tablet.cc:2369] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:17:03.925197 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.920437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.24.216.129:42021)
0929 02:17:03.920608 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:03.920612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:03.920628 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:17:03.920743 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:03.920746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:03.920829 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:03.920945 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:03.921239 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.921908 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.923323 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.923717 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.923805 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.924204 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:03.924666 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:03.924768 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:03.924848 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 02:17:03.924906 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:17:03.933058 11063 raft_consensus.cc:1261] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1 [term 1 FOLLOWER]: Refusing update from remote peer d60083dd50a34ce184e7763a5a4668c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:17:03.933125 10933 raft_consensus.cc:1261] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4 [term 1 FOLLOWER]: Refusing update from remote peer d60083dd50a34ce184e7763a5a4668c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:17:03.933921 11137 consensus_queue.cc:1035] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de1ac7dff4b47cd9d446f6309c309c4" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:17:03.934340 11133 consensus_queue.cc:1035] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3c5356d965f471cab09583564cb62a1" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:17:03.959980 10660 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:17:03.960282 10660 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:17:03.963738 10660 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b062215cf3947a9a3e24769bfe5be30: Sending DeleteTablet for 3 replicas of tablet 0103c7077f144d6fba478665f9855a85
I20260929 02:17:03.964838 10913 tablet_service.cc:1514] Processing DeleteTablet for tablet 0103c7077f144d6fba478665f9855a85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:03 UTC) from {username='slave'} at 127.0.0.1:42414
I20260929 02:17:03.965209 11043 tablet_service.cc:1514] Processing DeleteTablet for tablet 0103c7077f144d6fba478665f9855a85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:03 UTC) from {username='slave'} at 127.0.0.1:57372
I20260929 02:17:03.965969 10782 tablet_service.cc:1514] Processing DeleteTablet for tablet 0103c7077f144d6fba478665f9855a85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:03 UTC) from {username='slave'} at 127.0.0.1:42112
I20260929 02:17:03.968685 11153 tablet_replica.cc:330] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0: stopping tablet replica
I20260929 02:17:03.969137 11153 raft_consensus.cc:2229] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:17:03.969221 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 10720
I20260929 02:17:03.969919 11153 raft_consensus.cc:2258] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:03.973747 11153 ts_tablet_manager.cc:1905] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:03.981823 11153 ts_tablet_manager.cc:1918] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:03.982607 11153 log.cc:1196] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/0103c7077f144d6fba478665f9855a85
I20260929 02:17:03.983377 11153 ts_tablet_manager.cc:1939] T 0103c7077f144d6fba478665f9855a85 P d60083dd50a34ce184e7763a5a4668c0: Deleting consensus metadata
I20260929 02:17:03.987071 11152 tablet_replica.cc:330] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1: stopping tablet replica
I20260929 02:17:03.987542 11152 raft_consensus.cc:2229] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:17:03.988088 11152 raft_consensus.cc:2258] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:03.989054 10645 catalog_manager.cc:4841] TS d60083dd50a34ce184e7763a5a4668c0 (127.24.216.129:42021): tablet 0103c7077f144d6fba478665f9855a85 (table test-table [id=530dc304f4e347efbf6e4a8598b8fe9f]) successfully deleted
I20260929 02:17:03.990563 11152 ts_tablet_manager.cc:1905] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:03.991065 11151 tablet_replica.cc:330] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4: stopping tablet replica
I20260929 02:17:03.991480 11151 raft_consensus.cc:2229] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:17:03.992525 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:03.971370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.24.216.129:42021)
0929 02:17:03.971532 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:03.971538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:03.982144 (+ 10606us) server_negotiation.cc:407] Connection header received
0929 02:17:03.982447 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:03.982452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:03.982581 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:03.982668 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:03.983406 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.984627 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.988508 (+ 3881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:03.989258 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:03.989351 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:03.990872 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:03.990885 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:03.990902 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:03.991632 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:03.991705 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:03.991728 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:03.991835 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:03.992049 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 02:17:03.992159 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:17:03.992612 11151 raft_consensus.cc:2258] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:03.993216 10842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:03.994735 11151 ts_tablet_manager.cc:1905] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:04.000711 11151 ts_tablet_manager.cc:1918] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:04.000877 11151 log.cc:1196] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/0103c7077f144d6fba478665f9855a85
I20260929 02:17:04.001319 11151 ts_tablet_manager.cc:1939] T 0103c7077f144d6fba478665f9855a85 P 5de1ac7dff4b47cd9d446f6309c309c4: Deleting consensus metadata
I20260929 02:17:04.002638 11152 ts_tablet_manager.cc:1918] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:04.002794 11152 log.cc:1196] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/0103c7077f144d6fba478665f9855a85
I20260929 02:17:04.002894 10646 catalog_manager.cc:4841] TS 5de1ac7dff4b47cd9d446f6309c309c4 (127.24.216.130:33251): tablet 0103c7077f144d6fba478665f9855a85 (table test-table [id=530dc304f4e347efbf6e4a8598b8fe9f]) successfully deleted
I20260929 02:17:04.003337 11152 ts_tablet_manager.cc:1939] T 0103c7077f144d6fba478665f9855a85 P b3c5356d965f471cab09583564cb62a1: Deleting consensus metadata
I20260929 02:17:04.004935 10645 catalog_manager.cc:4841] TS b3c5356d965f471cab09583564cb62a1 (127.24.216.131:45913): tablet 0103c7077f144d6fba478665f9855a85 (table test-table [id=530dc304f4e347efbf6e4a8598b8fe9f]) successfully deleted
I20260929 02:17:04.090433 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 10720
I20260929 02:17:04.107384 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 10851
I20260929 02:17:04.113831 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:04.108127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.24.216.130:33251)
0929 02:17:04.108317 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:04.108324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:04.108346 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:17:04.108458 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:04.108462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:04.108550 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:04.108635 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:04.108987 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:04.109823 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:04.111184 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:04.111560 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:04.111649 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:04.112635 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:04.112643 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:04.112656 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:04.113220 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:04.113263 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:04.113281 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:04.113363 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:04.113497 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 02:17:04.113577 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:17:04.114534 10973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:04.203364 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 10851
I20260929 02:17:04.223136 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 10981
I20260929 02:17:04.235205 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:04.223849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.24.216.131:45913)
0929 02:17:04.224010 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:04.224014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:04.224082 (+ 68us) server_negotiation.cc:407] Connection header received
0929 02:17:04.224234 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:04.224238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:04.224324 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:04.224406 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:04.224752 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:04.225452 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:04.226743 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:04.227107 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:04.227207 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:04.233794 (+ 6587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:04.233803 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:04.233818 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:04.234457 (+ 639us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:04.234506 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:04.234527 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:04.234613 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:04.234762 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 02:17:04.234901 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:17:04.235834 11103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:04.323728 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 10981
I20260929 02:17:04.341572 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 10630
I20260929 02:17:04.342680 10691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:04.450616 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 10630
2026-09-29T02:17:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2759 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:04 dist-test-slave-9k0h krb5kdc[11161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:04 dist-test-slave-9k0h krb5kdc[11161](info): set up 2 sockets
Sep 29 02:17:04 dist-test-slave-9k0h krb5kdc[11161](info): commencing operation
krb5kdc: starting...
W20260929 02:17:05.590253 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:05 dist-test-slave-9k0h krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:17:05.846094 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:45813
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:37781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:45813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:06.165846 11174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:06.166213 11174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:06.166373 11174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:06.173614 11174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:06.173760 11174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:06.173871 11174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:06.173967 11174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:06.174046 11174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:06.183883 11174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:37781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:45813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:45813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:06.185253 11174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:06.186532 11174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:06.198102 11179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:06.198180 11182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:06.198180 11180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:06.199319 11174 server_base.cc:1034] running on GCE node
I20260929 02:17:06.200085 11174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:06.201443 11174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:06.202659 11174 hybrid_clock.cc:648] HybridClock initialized: now 1790648226202608 us; error 46 us; skew 500 ppm
I20260929 02:17:06.202979 11174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:06 dist-test-slave-9k0h krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648226, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:06.205928 11174 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:17:06.207458 11174 webserver.cc:460] Webserver started at http://127.24.216.190:43119/ using document root <none> and password file <none>
I20260929 02:17:06.208105 11174 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:06.208244 11174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:06.208576 11174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:06.211551 11174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "bbc103c85b8c405c8358d70aeeff04d7"
format_stamp: "Formatted at 2026-09-29 02:17:06 on dist-test-slave-9k0h"
server_key: "fe9862c80bf270683fdb698a4bb8eda0"
server_key_iv: "05131cf22b96331e4107365a2040ce17"
server_key_version: "encryptionkey@0"
I20260929 02:17:06.212280 11174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "bbc103c85b8c405c8358d70aeeff04d7"
format_stamp: "Formatted at 2026-09-29 02:17:06 on dist-test-slave-9k0h"
server_key: "fe9862c80bf270683fdb698a4bb8eda0"
server_key_iv: "05131cf22b96331e4107365a2040ce17"
server_key_version: "encryptionkey@0"
I20260929 02:17:06.217162 11174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:17:06.220477 11189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:06.221652 11174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:17:06.221822 11174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "bbc103c85b8c405c8358d70aeeff04d7"
format_stamp: "Formatted at 2026-09-29 02:17:06 on dist-test-slave-9k0h"
server_key: "fe9862c80bf270683fdb698a4bb8eda0"
server_key_iv: "05131cf22b96331e4107365a2040ce17"
server_key_version: "encryptionkey@0"
I20260929 02:17:06.222007 11174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:06.233435 11174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:06.237393 11174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:06.237659 11174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:06.254535 11174 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:45813
I20260929 02:17:06.254542 11240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:45813 every 8 connection(s)
I20260929 02:17:06.255903 11174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:06.259532 11241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:06.260712 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 11174
I20260929 02:17:06.260910 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:06.261474 25442 external_mini_cluster.cc:1428] Setting key d4b248e221d85a4215f143a06192c78a
I20260929 02:17:06.273974 11241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7: Bootstrap starting.
Sep 29 02:17:06 dist-test-slave-9k0h krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:17:06.277290 11241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:06.278512 11241 log.cc:824] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:06.281697 11241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7: No bootstrap required, opened a new log
I20260929 02:17:06.285241 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:06.263370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.24.216.190:45813)
0929 02:17:06.264154 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:06.264169 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:06.264228 (+ 59us) server_negotiation.cc:407] Connection header received
0929 02:17:06.266526 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:06.266555 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:06.267055 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:06.267485 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:06.268818 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:06.270839 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:06.272178 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:06.273199 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:06.273380 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:06.277910 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:06.277919 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:06.277934 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:06.277970 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:06.281075 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:06.281924 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:06.281930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:06.281936 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:06.282108 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:06.282584 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:06.282591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:06.282593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:06.283103 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:06.283317 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:06.283811 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 02:17:06.284219 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":363,"threads_started":1}
I20260929 02:17:06.290356 11241 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc103c85b8c405c8358d70aeeff04d7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45813 } }
I20260929 02:17:06.290724 11241 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:06.290843 11241 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc103c85b8c405c8358d70aeeff04d7, State: Initialized, Role: FOLLOWER
I20260929 02:17:06.291486 11241 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc103c85b8c405c8358d70aeeff04d7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45813 } }
I20260929 02:17:06.291695 11241 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:06.291788 11241 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:06.291908 11241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:06.293900 11241 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc103c85b8c405c8358d70aeeff04d7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45813 } }
I20260929 02:17:06.294396 11241 leader_election.cc:302] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbc103c85b8c405c8358d70aeeff04d7; no voters:
I20260929 02:17:06.294938 11241 leader_election.cc:288] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:06.295090 11246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:06.295764 11246 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [term 1 LEADER]: Becoming Leader. State: Replica: bbc103c85b8c405c8358d70aeeff04d7, State: Running, Role: LEADER
I20260929 02:17:06.296340 11246 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc103c85b8c405c8358d70aeeff04d7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45813 } }
I20260929 02:17:06.296986 11241 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:06.299364 11248 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbc103c85b8c405c8358d70aeeff04d7. Latest consensus state: current_term: 1 leader_uuid: "bbc103c85b8c405c8358d70aeeff04d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc103c85b8c405c8358d70aeeff04d7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45813 } } }
I20260929 02:17:06.299456 11247 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbc103c85b8c405c8358d70aeeff04d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc103c85b8c405c8358d70aeeff04d7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45813 } } }
I20260929 02:17:06.299685 11248 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:06.299814 11247 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:06.301445 11253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:06.307165 11253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:06.316223 11253 catalog_manager.cc:1347] Generated new cluster ID: 75e6043c153c499c854a2d5e1a99ecb4
I20260929 02:17:06.316344 11253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:06.354194 11253 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:06.355885 11253 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:06.365839 11253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7: Generated new TSK 0
I20260929 02:17:06.366901 11253 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:17:06.427778 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:45813
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:37781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:06.754887 11267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:06.755334 11267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:06.755491 11267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:06.762949 11267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:06.763108 11267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:06.763293 11267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:17:06.773319 11267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:37781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:45813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:06.774719 11267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:06.776124 11267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:06.789719 11272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:06.789608 11273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:06.789687 11275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:06.790571 11267 server_base.cc:1034] running on GCE node
I20260929 02:17:06.791301 11267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:06.792295 11267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:06.793545 11267 hybrid_clock.cc:648] HybridClock initialized: now 1790648226793482 us; error 53 us; skew 500 ppm
I20260929 02:17:06.793905 11267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:06 dist-test-slave-9k0h krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648226, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:06.797286 11267 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:17:06.798904 11267 webserver.cc:460] Webserver started at http://127.24.216.129:37533/ using document root <none> and password file <none>
I20260929 02:17:06.799569 11267 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:06.799674 11267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:06.799953 11267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:06.803141 11267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "a283e11fc8b54d2081448d7ce573120a"
format_stamp: "Formatted at 2026-09-29 02:17:06 on dist-test-slave-9k0h"
server_key: "6001df16ffefa41a16ccae0046df280a"
server_key_iv: "84059f1cdb63461f7b26d70c03c773d0"
server_key_version: "encryptionkey@0"
I20260929 02:17:06.803944 11267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "a283e11fc8b54d2081448d7ce573120a"
format_stamp: "Formatted at 2026-09-29 02:17:06 on dist-test-slave-9k0h"
server_key: "6001df16ffefa41a16ccae0046df280a"
server_key_iv: "84059f1cdb63461f7b26d70c03c773d0"
server_key_version: "encryptionkey@0"
I20260929 02:17:06.809371 11267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:17:06.812937 11282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:06.814432 11267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:17:06.814569 11267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "a283e11fc8b54d2081448d7ce573120a"
format_stamp: "Formatted at 2026-09-29 02:17:06 on dist-test-slave-9k0h"
server_key: "6001df16ffefa41a16ccae0046df280a"
server_key_iv: "84059f1cdb63461f7b26d70c03c773d0"
server_key_version: "encryptionkey@0"
I20260929 02:17:06.814738 11267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:06.832995 11267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:06.837379 11267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:06.837625 11267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:06.838533 11267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:06.839828 11267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:06.839901 11267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:06.839986 11267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:06.840039 11267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:06.872583 11267 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:36541
I20260929 02:17:06.872629 11394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:36541 every 8 connection(s)
I20260929 02:17:06.873872 11267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:17:06.874462 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 11267
I20260929 02:17:06.874653 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:17:06.875237 25442 external_mini_cluster.cc:1428] Setting key 4a2bf53cd5c58e303ce6842a6cf50220
Sep 29 02:17:06 dist-test-slave-9k0h krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648226, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:17:06.893888 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:06.876624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:40027 (local address 127.24.216.190:45813)
0929 02:17:06.877104 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:06.877110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:06.878273 (+ 1163us) server_negotiation.cc:407] Connection header received
0929 02:17:06.880073 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:06.880078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:06.880161 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:06.880295 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:06.881779 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:06.882634 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:06.884186 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:06.884616 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:06.884762 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:06.888129 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:06.888134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:06.888137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:06.888169 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:06.890810 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:06.891533 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:06.891542 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:06.891546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:06.891721 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:06.892143 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:06.892149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:06.892153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:06.892425 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:06.892577 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:06.893273 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 02:17:06.893505 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":236,"threads_started":1}
I20260929 02:17:06.894934 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:06.877349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:45813 (local address 127.24.216.129:40027)
0929 02:17:06.878081 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:06.878123 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:06.879822 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:06.880550 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:06.880558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:06.881060 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:06.881619 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:06.881635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:06.882814 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:06.882824 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:06.884042 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:06.884053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:06.884855 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:06.884864 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:06.885093 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:06.885763 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:06.885790 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:06.887914 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:06.890950 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:06.890960 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:06.890977 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:06.891414 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:06.891831 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:06.891836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:06.891840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:06.892034 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:06.892549 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:06.892557 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:06.892915 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 02:17:06.893180 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 02:17:06.893728 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":306,"threads_started":1}
I20260929 02:17:06.896492 11395 heartbeater.cc:347] Connected to a master server at 127.24.216.190:45813
I20260929 02:17:06.896853 11395 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:06.897603 11395 heartbeater.cc:511] Master 127.24.216.190:45813 requested a full tablet report, sending...
I20260929 02:17:06.899886 11206 ts_manager.cc:194] Registered new tserver with Master: a283e11fc8b54d2081448d7ce573120a (127.24.216.129:36541)
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
I20260929 02:17:06.901373 11206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:40027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:17:06.937089 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:45813
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:37781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:07.271431 11401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:07.271715 11401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:07.271879 11401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:07.279395 11401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:07.279512 11401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:07.279683 11401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:17:07.289628 11401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:37781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:45813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:07.290975 11401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:07.292322 11401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:07.305495 11406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:07.305486 11407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:07.305428 11409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:07.306362 11401 server_base.cc:1034] running on GCE node
I20260929 02:17:07.306988 11401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:07.307977 11401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:07.309201 11401 hybrid_clock.cc:648] HybridClock initialized: now 1790648227309152 us; error 42 us; skew 500 ppm
I20260929 02:17:07.309516 11401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:07 dist-test-slave-9k0h krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648227, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:07.312537 11401 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:17:07.314018 11401 webserver.cc:460] Webserver started at http://127.24.216.130:34985/ using document root <none> and password file <none>
I20260929 02:17:07.314744 11401 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:07.314893 11401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:07.315215 11401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:07.318207 11401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "9c6a4abd243b415e88fa218e87ab1211"
format_stamp: "Formatted at 2026-09-29 02:17:07 on dist-test-slave-9k0h"
server_key: "3fe1b3fec32c1b1bea77f103c64250b4"
server_key_iv: "2c4d6306c244173705ceecac343bfdf0"
server_key_version: "encryptionkey@0"
I20260929 02:17:07.319012 11401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "9c6a4abd243b415e88fa218e87ab1211"
format_stamp: "Formatted at 2026-09-29 02:17:07 on dist-test-slave-9k0h"
server_key: "3fe1b3fec32c1b1bea77f103c64250b4"
server_key_iv: "2c4d6306c244173705ceecac343bfdf0"
server_key_version: "encryptionkey@0"
I20260929 02:17:07.323962 11401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:17:07.327762 11416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:07.329046 11401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:17:07.329211 11401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "9c6a4abd243b415e88fa218e87ab1211"
format_stamp: "Formatted at 2026-09-29 02:17:07 on dist-test-slave-9k0h"
server_key: "3fe1b3fec32c1b1bea77f103c64250b4"
server_key_iv: "2c4d6306c244173705ceecac343bfdf0"
server_key_version: "encryptionkey@0"
I20260929 02:17:07.329424 11401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:07.342052 11401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:07.345929 11401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:07.346248 11401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:07.347169 11401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:07.348526 11401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:07.348621 11401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:07.348762 11401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:07.348835 11401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:07.384500 11401 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:39731
I20260929 02:17:07.384522 11528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:39731 every 8 connection(s)
I20260929 02:17:07.385917 11401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:17:07.389317 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 11401
I20260929 02:17:07.389513 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:17:07.390060 25442 external_mini_cluster.cc:1428] Setting key 15cb99d4e9063131c05ddb29ec687a9e
Sep 29 02:17:07 dist-test-slave-9k0h krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648227, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:17:07.409085 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:07.388780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:53597 (local address 127.24.216.190:45813)
0929 02:17:07.389019 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:07.389028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:07.390521 (+ 1493us) server_negotiation.cc:407] Connection header received
0929 02:17:07.392762 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:07.392769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:07.392869 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:07.392995 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:07.394746 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.395501 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.397200 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.397743 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.397890 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:07.402087 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:07.402092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:07.402095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:07.402128 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:07.405800 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:07.406652 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:07.406656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:07.406658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:07.406797 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:07.407266 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:07.407269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:07.407271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:07.407538 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:07.407693 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:07.408482 (+ 789us) server_negotiation.cc:299] Negotiation successful
0929 02:17:07.408728 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:17:07.410107 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:07.389359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:45813 (local address 127.24.216.130:53597)
0929 02:17:07.390323 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:07.390382 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:07.392475 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:07.393257 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:07.393265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:07.393818 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:07.394560 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:07.394579 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.395662 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.395668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:07.397049 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:07.397062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.397992 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.398002 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:07.398399 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:07.399216 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:07.399239 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:07.401854 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:07.405963 (+ 4109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:07.405973 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:07.405992 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:07.406509 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:07.406913 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:07.406918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:07.406922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:07.407126 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:07.407670 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:07.407680 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:07.408079 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 02:17:07.408375 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 02:17:07.408960 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":403,"threads_started":1}
I20260929 02:17:07.411628 11529 heartbeater.cc:347] Connected to a master server at 127.24.216.190:45813
I20260929 02:17:07.411999 11529 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:07.413007 11529 heartbeater.cc:511] Master 127.24.216.190:45813 requested a full tablet report, sending...
I20260929 02:17:07.414789 11206 ts_manager.cc:194] Registered new tserver with Master: 9c6a4abd243b415e88fa218e87ab1211 (127.24.216.130:39731)
I20260929 02:17:07.416224 11206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:53597
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:17:07.461431 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:45813
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:37781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:07.795145 11534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:07.795542 11534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:07.795708 11534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:07.804924 11534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:07.805025 11534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:07.805148 11534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:17:07.815352 11534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:37781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:45813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:07.816805 11534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:07.818280 11534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:07.831346 11539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:07.831436 11540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:07.831368 11542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:07.832122 11534 server_base.cc:1034] running on GCE node
I20260929 02:17:07.832778 11534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:07.833730 11534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:07.834945 11534 hybrid_clock.cc:648] HybridClock initialized: now 1790648227834900 us; error 30 us; skew 500 ppm
I20260929 02:17:07.835287 11534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:07 dist-test-slave-9k0h krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648227, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:07.838351 11534 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:17:07.839699 11534 webserver.cc:460] Webserver started at http://127.24.216.131:40843/ using document root <none> and password file <none>
I20260929 02:17:07.840368 11534 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:07.840505 11534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:07.840808 11534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:07.843720 11534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "4058c0c8c11b4392b4c77a1f065c240a"
format_stamp: "Formatted at 2026-09-29 02:17:07 on dist-test-slave-9k0h"
server_key: "474f1c9806d90bae5e41be14d3349d9e"
server_key_iv: "f2841f8e51fa381ba66b7f39ad039838"
server_key_version: "encryptionkey@0"
I20260929 02:17:07.844470 11534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "4058c0c8c11b4392b4c77a1f065c240a"
format_stamp: "Formatted at 2026-09-29 02:17:07 on dist-test-slave-9k0h"
server_key: "474f1c9806d90bae5e41be14d3349d9e"
server_key_iv: "f2841f8e51fa381ba66b7f39ad039838"
server_key_version: "encryptionkey@0"
I20260929 02:17:07.849253 11534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:17:07.852411 11549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:07.853564 11534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:17:07.853729 11534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "4058c0c8c11b4392b4c77a1f065c240a"
format_stamp: "Formatted at 2026-09-29 02:17:07 on dist-test-slave-9k0h"
server_key: "474f1c9806d90bae5e41be14d3349d9e"
server_key_iv: "f2841f8e51fa381ba66b7f39ad039838"
server_key_version: "encryptionkey@0"
I20260929 02:17:07.853910 11534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:07.870055 11534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:07.874889 11534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:07.875212 11534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:07.876305 11534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:07.877920 11534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:07.878049 11534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:07.878197 11534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:07.878265 11534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 02:17:07.904605 11395 heartbeater.cc:503] Master 127.24.216.190:45813 was elected leader, sending a full tablet report...
I20260929 02:17:07.909466 11534 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:37105
I20260929 02:17:07.909487 11661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:37105 every 8 connection(s)
I20260929 02:17:07.910826 11534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:17:07.918481 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 11534
I20260929 02:17:07.918679 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:17:07.919281 25442 external_mini_cluster.cc:1428] Setting key 6d6536b22cf32184746b943ef91eb7b4
Sep 29 02:17:07 dist-test-slave-9k0h krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648227, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:17:07.930342 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:07.913175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:60887 (local address 127.24.216.190:45813)
0929 02:17:07.913561 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:07.913566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:07.914650 (+ 1084us) server_negotiation.cc:407] Connection header received
0929 02:17:07.916411 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:07.916418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:07.916498 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:07.916605 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:07.918033 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.918932 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.920812 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.921336 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.921465 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:07.924690 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:07.924705 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:07.924710 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:07.924748 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:07.927386 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:07.928048 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:07.928056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:07.928061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:07.928181 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:07.928624 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:07.928630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:07.928634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:07.928901 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:07.929053 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:07.929718 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 02:17:07.929932 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":194,"threads_started":1}
I20260929 02:17:07.931238 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:07.913803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:45813 (local address 127.24.216.131:60887)
0929 02:17:07.914508 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:07.914545 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:07.916139 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:07.916836 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:07.916845 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:07.917334 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:07.917856 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:07.917872 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.919070 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.919081 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:07.920669 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:07.920681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.921552 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.921558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:07.921789 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:07.922464 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:07.922490 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:07.924504 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:07.927513 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:07.927521 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:07.927535 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:07.927934 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:07.928283 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:07.928289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:07.928292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:07.928520 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:07.929037 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:07.929044 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:07.929369 (+ 325us) client_negotiation.cc:769] Sending connection context
0929 02:17:07.929626 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 02:17:07.930100 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":305,"threads_started":1}
I20260929 02:17:07.932464 11662 heartbeater.cc:347] Connected to a master server at 127.24.216.190:45813
I20260929 02:17:07.932797 11662 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:07.933663 11662 heartbeater.cc:511] Master 127.24.216.190:45813 requested a full tablet report, sending...
I20260929 02:17:07.935127 11206 ts_manager.cc:194] Registered new tserver with Master: 4058c0c8c11b4392b4c77a1f065c240a (127.24.216.131:37105)
I20260929 02:17:07.935950 11206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:60887
I20260929 02:17:07.946249 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:17:07.962904 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:07.950123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.24.216.190:45813)
0929 02:17:07.950359 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:07.950368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:07.950668 (+ 300us) server_negotiation.cc:407] Connection header received
0929 02:17:07.950856 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:07.950861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:07.950945 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:07.951044 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:07.952538 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.953248 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.954679 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.955089 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.955214 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:07.957836 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:07.957843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:07.957849 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:07.957889 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:07.960227 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:07.960910 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:07.960916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:07.960919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:07.961032 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:07.961451 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:07.961456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:07.961460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:07.961756 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:07.962025 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:07.962448 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 02:17:07.962630 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 02:17:07.966470 11206 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:17:07.973246 11206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:17:07.996129 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:07.984788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:37105 (local address 127.0.0.1:39100)
0929 02:17:07.986009 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:07.986051 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:07.986531 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:07.988281 (+ 1750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:07.988289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:07.988325 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:07.988774 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:07.988790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.991007 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.991013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:07.993306 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:07.993320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.995171 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.995178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:07.995408 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:07.995471 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 02:17:07.995567 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 02:17:07.995745 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"thread_start_us":835,"threads_started":1}
I20260929 02:17:07.998220 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:07.985056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.24.216.131:37105)
0929 02:17:07.987626 (+ 2570us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:07.987634 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:07.987669 (+ 35us) server_negotiation.cc:407] Connection header received
0929 02:17:07.987767 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:07.987775 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:07.988039 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:07.988991 (+ 952us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:07.989260 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.990862 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.993511 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.994938 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.997384 (+ 2446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:07.997568 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:07.997636 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 02:17:07.997800 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2399,"thread_start_us":2226,"threads_started":1}
I20260929 02:17:07.999716 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:07.985246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.24.216.130:39731)
0929 02:17:07.987214 (+ 1968us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:07.987227 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:07.987736 (+ 509us) server_negotiation.cc:407] Connection header received
0929 02:17:07.988000 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:07.988007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:07.988223 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:07.988370 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:07.990395 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.991923 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.995397 (+ 3474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.998357 (+ 2960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.998486 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:07.998651 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:07.999216 (+ 565us) server_negotiation.cc:299] Negotiation successful
0929 02:17:07.999385 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1813,"thread_start_us":1683,"threads_started":1}
I20260929 02:17:08.000375 11676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:07.984991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:39731 (local address 127.0.0.1:44192)
0929 02:17:07.987614 (+ 2623us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:07.987633 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:07.987831 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:07.989730 (+ 1899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:07.989735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:07.989759 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:07.990221 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:07.990234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.993070 (+ 2836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.993075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:07.995224 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:07.995236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:07.998853 (+ 3617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:07.998859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:07.999026 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:07.999056 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:17:07.999909 (+ 853us) client_negotiation.cc:240] Negotiation successful
0929 02:17:08.000039 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2514,"thread_start_us":2381,"threads_started":1}
I20260929 02:17:08.002563 11597 tablet_service.cc:1467] Processing CreateTablet for tablet f62f803be58e4c2eb6f07b9a02b38250 (DEFAULT_TABLE table=test-table [id=03f0904c08624d32b2bf191f4733f263]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:08.003883 11597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62f803be58e4c2eb6f07b9a02b38250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:08.004772 11464 tablet_service.cc:1467] Processing CreateTablet for tablet f62f803be58e4c2eb6f07b9a02b38250 (DEFAULT_TABLE table=test-table [id=03f0904c08624d32b2bf191f4733f263]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:08.005832 11464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62f803be58e4c2eb6f07b9a02b38250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:08.008631 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:07.987730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:36541 (local address 127.0.0.1:33420)
0929 02:17:07.996392 (+ 8662us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:07.996411 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:07.996502 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:08.001408 (+ 4906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:08.001413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:08.001440 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:08.001615 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:08.001631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.004309 (+ 2678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.004314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:08.006039 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:08.006056 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.007875 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.007882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:08.008070 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:08.008105 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:17:08.008197 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 02:17:08.008340 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8533,"thread_start_us":13579,"threads_started":1}
I20260929 02:17:08.009388 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:07.988071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.24.216.129:36541)
0929 02:17:08.000664 (+ 12593us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:08.000677 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:08.000714 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:17:08.000833 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:08.000845 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:08.001155 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:08.001758 (+ 603us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:08.002034 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.004162 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.006226 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.007681 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.007881 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:08.008041 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:08.008901 (+ 860us) server_negotiation.cc:299] Negotiation successful
0929 02:17:08.009068 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12347,"thread_start_us":12866,"threads_started":1}
I20260929 02:17:08.013149 11330 tablet_service.cc:1467] Processing CreateTablet for tablet f62f803be58e4c2eb6f07b9a02b38250 (DEFAULT_TABLE table=test-table [id=03f0904c08624d32b2bf191f4733f263]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:08.014560 11330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62f803be58e4c2eb6f07b9a02b38250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:08.017505 11681 tablet_bootstrap.cc:492] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a: Bootstrap starting.
I20260929 02:17:08.019232 11682 tablet_bootstrap.cc:492] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211: Bootstrap starting.
I20260929 02:17:08.021248 11681 tablet_bootstrap.cc:654] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:08.022269 11681 log.cc:824] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:08.022627 11682 tablet_bootstrap.cc:654] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:08.023883 11682 log.cc:824] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:08.025899 11681 tablet_bootstrap.cc:492] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a: No bootstrap required, opened a new log
I20260929 02:17:08.026213 11681 ts_tablet_manager.cc:1397] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:17:08.027190 11682 tablet_bootstrap.cc:492] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211: No bootstrap required, opened a new log
I20260929 02:17:08.027447 11682 ts_tablet_manager.cc:1397] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 02:17:08.028978 11685 tablet_bootstrap.cc:492] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a: Bootstrap starting.
I20260929 02:17:08.032459 11685 tablet_bootstrap.cc:654] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:08.033628 11685 log.cc:824] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:08.035575 11682 raft_consensus.cc:348] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4058c0c8c11b4392b4c77a1f065c240a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 37105 } } peers { permanent_uuid: "9c6a4abd243b415e88fa218e87ab1211" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 39731 } } peers { permanent_uuid: "a283e11fc8b54d2081448d7ce573120a" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 36541 } }
I20260929 02:17:08.035967 11682 raft_consensus.cc:374] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:08.036065 11682 raft_consensus.cc:729] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6a4abd243b415e88fa218e87ab1211, State: Initialized, Role: FOLLOWER
I20260929 02:17:08.036685 11682 consensus_queue.cc:260] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4058c0c8c11b4392b4c77a1f065c240a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 37105 } } peers { permanent_uuid: "9c6a4abd243b415e88fa218e87ab1211" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 39731 } } peers { permanent_uuid: "a283e11fc8b54d2081448d7ce573120a" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 36541 } }
I20260929 02:17:08.036854 11685 tablet_bootstrap.cc:492] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a: No bootstrap required, opened a new log
I20260929 02:17:08.037096 11685 ts_tablet_manager.cc:1397] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 02:17:08.037124 11681 raft_consensus.cc:348] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4058c0c8c11b4392b4c77a1f065c240a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 37105 } } peers { permanent_uuid: "9c6a4abd243b415e88fa218e87ab1211" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 39731 } } peers { permanent_uuid: "a283e11fc8b54d2081448d7ce573120a" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 36541 } }
I20260929 02:17:08.037747 11529 heartbeater.cc:503] Master 127.24.216.190:45813 was elected leader, sending a full tablet report...
I20260929 02:17:08.038277 11682 ts_tablet_manager.cc:1428] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260929 02:17:08.038280 11681 raft_consensus.cc:374] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:08.038381 11681 raft_consensus.cc:729] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4058c0c8c11b4392b4c77a1f065c240a, State: Initialized, Role: FOLLOWER
I20260929 02:17:08.038892 11681 consensus_queue.cc:260] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4058c0c8c11b4392b4c77a1f065c240a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 37105 } } peers { permanent_uuid: "9c6a4abd243b415e88fa218e87ab1211" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 39731 } } peers { permanent_uuid: "a283e11fc8b54d2081448d7ce573120a" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 36541 } }
I20260929 02:17:08.039976 11662 heartbeater.cc:503] Master 127.24.216.190:45813 was elected leader, sending a full tablet report...
I20260929 02:17:08.040521 11681 ts_tablet_manager.cc:1428] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260929 02:17:08.046183 11685 raft_consensus.cc:348] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4058c0c8c11b4392b4c77a1f065c240a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 37105 } } peers { permanent_uuid: "9c6a4abd243b415e88fa218e87ab1211" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 39731 } } peers { permanent_uuid: "a283e11fc8b54d2081448d7ce573120a" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 36541 } }
I20260929 02:17:08.046541 11685 raft_consensus.cc:374] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:08.046655 11685 raft_consensus.cc:729] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a283e11fc8b54d2081448d7ce573120a, State: Initialized, Role: FOLLOWER
I20260929 02:17:08.047211 11685 consensus_queue.cc:260] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4058c0c8c11b4392b4c77a1f065c240a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 37105 } } peers { permanent_uuid: "9c6a4abd243b415e88fa218e87ab1211" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 39731 } } peers { permanent_uuid: "a283e11fc8b54d2081448d7ce573120a" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 36541 } }
I20260929 02:17:08.048836 11685 ts_tablet_manager.cc:1428] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260929 02:17:08.053262 11687 raft_consensus.cc:482] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:17:08.053516 11687 raft_consensus.cc:504] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4058c0c8c11b4392b4c77a1f065c240a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 37105 } } peers { permanent_uuid: "9c6a4abd243b415e88fa218e87ab1211" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 39731 } } peers { permanent_uuid: "a283e11fc8b54d2081448d7ce573120a" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 36541 } }
I20260929 02:17:08.055090 11687 leader_election.cc:288] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4058c0c8c11b4392b4c77a1f065c240a (127.24.216.131:37105), a283e11fc8b54d2081448d7ce573120a (127.24.216.129:36541)
I20260929 02:17:08.064510 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.055520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:38685 (local address 127.24.216.129:36541)
0929 02:17:08.055785 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:08.055792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:08.056462 (+ 670us) server_negotiation.cc:407] Connection header received
0929 02:17:08.056663 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:08.056668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:08.056736 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:08.056818 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:08.057749 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.058951 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.062007 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.063263 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.063424 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:08.063498 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:08.064124 (+ 626us) server_negotiation.cc:299] Negotiation successful
0929 02:17:08.064266 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 02:17:08.064549 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.055550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:33967 (local address 127.24.216.131:37105)
0929 02:17:08.055751 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:08.055759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:08.056408 (+ 649us) server_negotiation.cc:407] Connection header received
0929 02:17:08.056623 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:08.056629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:08.056713 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:08.056818 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:08.057712 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.058973 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.062008 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.063266 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.063424 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:08.063498 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:08.064156 (+ 658us) server_negotiation.cc:299] Negotiation successful
0929 02:17:08.064287 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:17:08.064669 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.055614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:36541 (local address 127.24.216.130:38685)
0929 02:17:08.056376 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:08.056391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:08.056478 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:08.056855 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:08.056860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:08.056912 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:08.057599 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:08.057612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.059123 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.059129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:08.061858 (+ 2729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:08.061866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.063517 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.063525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:08.063833 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:08.063988 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 02:17:08.064099 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 02:17:08.064246 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":376,"spinlock_wait_cycles":26880,"thread_start_us":440,"threads_started":1}
I20260929 02:17:08.065364 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.055402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:37105 (local address 127.24.216.130:33967)
0929 02:17:08.056301 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:08.056322 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:08.056474 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:08.056855 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:08.056860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:08.056913 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:08.057533 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:08.057551 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.059123 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.059129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:08.061849 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:08.061865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.063527 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.063532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:08.063829 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:08.063988 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 02:17:08.064813 (+ 825us) client_negotiation.cc:240] Negotiation successful
0929 02:17:08.064964 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":285,"spinlock_wait_cycles":8704,"thread_start_us":390,"threads_started":1}
I20260929 02:17:08.065788 11350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62f803be58e4c2eb6f07b9a02b38250" candidate_uuid: "9c6a4abd243b415e88fa218e87ab1211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a283e11fc8b54d2081448d7ce573120a" is_pre_election: true
I20260929 02:17:08.066390 11350 raft_consensus.cc:2454] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c6a4abd243b415e88fa218e87ab1211 in term 0.
I20260929 02:17:08.066537 11617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62f803be58e4c2eb6f07b9a02b38250" candidate_uuid: "9c6a4abd243b415e88fa218e87ab1211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4058c0c8c11b4392b4c77a1f065c240a" is_pre_election: true
I20260929 02:17:08.066988 11617 raft_consensus.cc:2454] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c6a4abd243b415e88fa218e87ab1211 in term 0.
I20260929 02:17:08.067014 11418 leader_election.cc:302] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c6a4abd243b415e88fa218e87ab1211, a283e11fc8b54d2081448d7ce573120a; no voters:
I20260929 02:17:08.067436 11687 raft_consensus.cc:2781] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:17:08.067595 11687 raft_consensus.cc:482] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:17:08.067698 11687 raft_consensus.cc:3037] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:08.069890 11687 raft_consensus.cc:504] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4058c0c8c11b4392b4c77a1f065c240a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 37105 } } peers { permanent_uuid: "9c6a4abd243b415e88fa218e87ab1211" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 39731 } } peers { permanent_uuid: "a283e11fc8b54d2081448d7ce573120a" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 36541 } }
I20260929 02:17:08.070494 11687 leader_election.cc:288] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [CANDIDATE]: Term 1 election: Requested vote from peers 4058c0c8c11b4392b4c77a1f065c240a (127.24.216.131:37105), a283e11fc8b54d2081448d7ce573120a (127.24.216.129:36541)
I20260929 02:17:08.070963 11617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62f803be58e4c2eb6f07b9a02b38250" candidate_uuid: "9c6a4abd243b415e88fa218e87ab1211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4058c0c8c11b4392b4c77a1f065c240a"
I20260929 02:17:08.070963 11350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62f803be58e4c2eb6f07b9a02b38250" candidate_uuid: "9c6a4abd243b415e88fa218e87ab1211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a283e11fc8b54d2081448d7ce573120a"
I20260929 02:17:08.071165 11617 raft_consensus.cc:3037] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:08.071167 11350 raft_consensus.cc:3037] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:08.074357 11617 raft_consensus.cc:2454] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c6a4abd243b415e88fa218e87ab1211 in term 1.
I20260929 02:17:08.074357 11350 raft_consensus.cc:2454] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c6a4abd243b415e88fa218e87ab1211 in term 1.
I20260929 02:17:08.074755 11417 leader_election.cc:302] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4058c0c8c11b4392b4c77a1f065c240a, 9c6a4abd243b415e88fa218e87ab1211, a283e11fc8b54d2081448d7ce573120a; no voters:
I20260929 02:17:08.075088 11687 raft_consensus.cc:2781] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:08.075529 11687 raft_consensus.cc:686] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [term 1 LEADER]: Becoming Leader. State: Replica: 9c6a4abd243b415e88fa218e87ab1211, State: Running, Role: LEADER
I20260929 02:17:08.075991 11687 consensus_queue.cc:237] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4058c0c8c11b4392b4c77a1f065c240a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 37105 } } peers { permanent_uuid: "9c6a4abd243b415e88fa218e87ab1211" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 39731 } } peers { permanent_uuid: "a283e11fc8b54d2081448d7ce573120a" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 36541 } }
I20260929 02:17:08.080530 11206 catalog_manager.cc:5495] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c6a4abd243b415e88fa218e87ab1211 (127.24.216.130). New cstate: current_term: 1 leader_uuid: "9c6a4abd243b415e88fa218e87ab1211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4058c0c8c11b4392b4c77a1f065c240a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 37105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c6a4abd243b415e88fa218e87ab1211" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 39731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a283e11fc8b54d2081448d7ce573120a" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 36541 } health_report { overall_health: UNKNOWN } } }
I20260929 02:17:08.095532 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.090457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44194 (local address 127.24.216.130:39731)
0929 02:17:08.090658 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:08.090666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:08.090751 (+ 85us) server_negotiation.cc:407] Connection header received
0929 02:17:08.090912 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:08.090916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:08.090993 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:08.091108 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:08.091443 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.092148 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.093449 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.093847 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.093936 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:08.094443 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:08.094919 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:08.095042 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:08.095188 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 02:17:08.095227 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:17:08.103279 11350 raft_consensus.cc:1261] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a [term 1 FOLLOWER]: Refusing update from remote peer 9c6a4abd243b415e88fa218e87ab1211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:17:08.103296 11617 raft_consensus.cc:1261] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a [term 1 FOLLOWER]: Refusing update from remote peer 9c6a4abd243b415e88fa218e87ab1211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:17:08.104230 11692 consensus_queue.cc:1035] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4058c0c8c11b4392b4c77a1f065c240a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 37105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:17:08.104578 11687 consensus_queue.cc:1035] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a283e11fc8b54d2081448d7ce573120a" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 36541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:17:08.116281 11696 mvcc.cc:205] Tried to move back new op lower bound from 7334495142301503488 to 7334495142203359232. Current Snapshot: MvccSnapshot[applied={T|T < 7334495142301503488}]
I20260929 02:17:08.134809 11206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:17:08.135069 11206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:17:08.138522 11206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbc103c85b8c405c8358d70aeeff04d7: Sending DeleteTablet for 3 replicas of tablet f62f803be58e4c2eb6f07b9a02b38250
I20260929 02:17:08.139748 11464 tablet_service.cc:1514] Processing DeleteTablet for tablet f62f803be58e4c2eb6f07b9a02b38250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:08 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:44192
I20260929 02:17:08.139981 11597 tablet_service.cc:1514] Processing DeleteTablet for tablet f62f803be58e4c2eb6f07b9a02b38250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:08 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:39100
I20260929 02:17:08.140599 11330 tablet_service.cc:1514] Processing DeleteTablet for tablet f62f803be58e4c2eb6f07b9a02b38250 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:08 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:33420
I20260929 02:17:08.140957 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 11267
I20260929 02:17:08.151902 11706 tablet_replica.cc:330] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a: stopping tablet replica
I20260929 02:17:08.152558 11707 tablet_replica.cc:330] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a: stopping tablet replica
I20260929 02:17:08.153051 11707 raft_consensus.cc:2229] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:17:08.153395 11705 tablet_replica.cc:330] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211: stopping tablet replica
I20260929 02:17:08.153699 11706 raft_consensus.cc:2229] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:17:08.153771 11705 raft_consensus.cc:2229] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:17:08.154352 11707 raft_consensus.cc:2258] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:08.154527 11705 raft_consensus.cc:2258] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:08.154804 11706 raft_consensus.cc:2258] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:08.156805 11705 ts_tablet_manager.cc:1905] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:08.158823 11706 ts_tablet_manager.cc:1905] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:08.159502 11707 ts_tablet_manager.cc:1905] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:17:08 dist-test-slave-9k0h krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:17:08.164809 11705 ts_tablet_manager.cc:1918] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:08.165061 11705 log.cc:1196] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/f62f803be58e4c2eb6f07b9a02b38250
I20260929 02:17:08.165694 11705 ts_tablet_manager.cc:1939] T f62f803be58e4c2eb6f07b9a02b38250 P 9c6a4abd243b415e88fa218e87ab1211: Deleting consensus metadata
I20260929 02:17:08.167610 11191 catalog_manager.cc:4841] TS 9c6a4abd243b415e88fa218e87ab1211 (127.24.216.130:39731): tablet f62f803be58e4c2eb6f07b9a02b38250 (table test-table [id=03f0904c08624d32b2bf191f4733f263]) successfully deleted
I20260929 02:17:08.168453 11706 ts_tablet_manager.cc:1918] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:08.168689 11706 log.cc:1196] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/f62f803be58e4c2eb6f07b9a02b38250
I20260929 02:17:08.169277 11706 ts_tablet_manager.cc:1939] T f62f803be58e4c2eb6f07b9a02b38250 P 4058c0c8c11b4392b4c77a1f065c240a: Deleting consensus metadata
I20260929 02:17:08.171128 11190 catalog_manager.cc:4841] TS 4058c0c8c11b4392b4c77a1f065c240a (127.24.216.131:37105): tablet f62f803be58e4c2eb6f07b9a02b38250 (table test-table [id=03f0904c08624d32b2bf191f4733f263]) successfully deleted
I20260929 02:17:08.172845 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.142569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.24.216.129:36541)
0929 02:17:08.142796 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:08.142806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:08.152772 (+ 9966us) server_negotiation.cc:407] Connection header received
0929 02:17:08.152973 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:08.152980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:08.153068 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:08.153182 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:08.155029 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.156230 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.158588 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.159256 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.159352 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:08.165056 (+ 5704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:08.165071 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:08.165089 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:08.165130 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:08.169122 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:08.170122 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:08.170131 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:08.170135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:08.170315 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:08.170797 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:08.170803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:08.170806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:08.171311 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:08.171609 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:08.172211 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 02:17:08.172456 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:17:08.173496 11707 ts_tablet_manager.cc:1918] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:08.173606 11390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:08.173682 11707 log.cc:1196] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/f62f803be58e4c2eb6f07b9a02b38250
I20260929 02:17:08.263742 11707 ts_tablet_manager.cc:1939] T f62f803be58e4c2eb6f07b9a02b38250 P a283e11fc8b54d2081448d7ce573120a: Deleting consensus metadata
I20260929 02:17:08.263772 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 11267
W20260929 02:17:08.278345 11191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.216.129:36541 (error 108)
I20260929 02:17:08.279162 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 11401
I20260929 02:17:08.283149 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.282658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:36541 (local address 127.0.0.1:33440)
0929 02:17:08.282845 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:08.282967 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 02:17:08.283447 11191 catalog_manager.cc:4584] TS a283e11fc8b54d2081448d7ce573120a (127.24.216.129:36541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f62f803be58e4c2eb6f07b9a02b38250: Network error: Client connection negotiation failed: client connection to 127.24.216.129:36541: connect: Connection refused (error 111)
Sep 29 02:17:08 dist-test-slave-9k0h krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:17:08.297013 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.281036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.24.216.130:39731)
0929 02:17:08.281278 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:08.281286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:08.281310 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:17:08.281391 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:08.281398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:08.281499 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:08.281642 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:08.283516 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.284758 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.286019 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.286700 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.286843 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:08.291213 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:08.291227 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:08.291246 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:08.291281 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:08.294324 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:08.295092 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:08.295099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:08.295101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:08.295228 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:08.295605 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:08.295609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:08.295611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:08.295947 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:08.296177 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:08.296516 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 02:17:08.296716 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:17:08.297808 11524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:08.337283 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.336823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:36541 (local address 127.0.0.1:33442)
0929 02:17:08.337043 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:08.337136 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 02:17:08.338317 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.337944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:36541 (local address 127.0.0.1:33454)
0929 02:17:08.338060 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:08.338121 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 02:17:08.398676 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.398255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:36541 (local address 127.0.0.1:33460)
0929 02:17:08.398436 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:08.398535 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 02:17:08.399641 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.399292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:36541 (local address 127.0.0.1:33464)
0929 02:17:08.399431 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:08.399508 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 02:17:08.414207 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 11401
I20260929 02:17:08.434545 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 11534
Sep 29 02:17:08 dist-test-slave-9k0h krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:17:08.450049 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.435275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.24.216.131:37105)
0929 02:17:08.435433 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:08.435438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:08.435516 (+ 78us) server_negotiation.cc:407] Connection header received
0929 02:17:08.435668 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:08.435671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:08.435741 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:08.435822 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:08.437070 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.437781 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.439485 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:08.440134 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:08.440269 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:08.445041 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:08.445056 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:08.445072 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:08.445110 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:08.447715 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:08.448347 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:08.448354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:08.448356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:08.448493 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:08.448813 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:08.448816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:08.448818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:08.449131 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:08.449351 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:08.449654 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 02:17:08.449813 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:17:08.450692 11657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:08.480739 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.480187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:36541 (local address 127.0.0.1:33476)
0929 02:17:08.480432 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:08.480546 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 02:17:08.481918 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.481444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:36541 (local address 127.0.0.1:33478)
0929 02:17:08.481654 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:08.481767 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 02:17:08.539695 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 11534
I20260929 02:17:08.559233 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 11174
I20260929 02:17:08.560385 11236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:08.667693 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:08.667008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:36541 (local address 127.0.0.1:33486)
0929 02:17:08.667321 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:08.667482 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260929 02:17:08.667774 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 11174
2026-09-29T02:17:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4214 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T02:17:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:08Z Disabled control of system clock
I20260929 02:17:08.741292 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:41587
--webserver_interface=127.24.216.190
--webserver_port=0
--builtin_ntp_servers=127.24.216.148:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:41587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:17:09.061265 11717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:09.061535 11717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:09.061671 11717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:09.068706 11717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:09.068814 11717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:09.068859 11717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:09.068915 11717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:09.068958 11717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:09.077860 11717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:41587
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:41587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:09.079114 11717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:09.080817 11717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:09.092226 11722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:09.092263 11725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:09.092286 11723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:09.093034 11717 server_base.cc:1034] running on GCE node
I20260929 02:17:09.093775 11717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:09.095124 11717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:09.096323 11717 hybrid_clock.cc:648] HybridClock initialized: now 1790648229096255 us; error 50 us; skew 500 ppm
I20260929 02:17:09.096664 11717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:17:09.098898 11717 webserver.cc:460] Webserver started at http://127.24.216.190:34865/ using document root <none> and password file <none>
I20260929 02:17:09.099546 11717 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:09.099632 11717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:09.099980 11717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:09.102972 11717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "99a7ecec7bca4213ae2038acc2760f90"
format_stamp: "Formatted at 2026-09-29 02:17:09 on dist-test-slave-9k0h"
server_key: "ddcb4950d607b8246044c7c856fb1191"
server_key_iv: "c1bd5c97973cf2e7bc23169989469a09"
server_key_version: "encryptionkey@0"
I20260929 02:17:09.103756 11717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "99a7ecec7bca4213ae2038acc2760f90"
format_stamp: "Formatted at 2026-09-29 02:17:09 on dist-test-slave-9k0h"
server_key: "ddcb4950d607b8246044c7c856fb1191"
server_key_iv: "c1bd5c97973cf2e7bc23169989469a09"
server_key_version: "encryptionkey@0"
I20260929 02:17:09.108560 11717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:17:09.111797 11731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:09.113040 11717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:17:09.113198 11717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "99a7ecec7bca4213ae2038acc2760f90"
format_stamp: "Formatted at 2026-09-29 02:17:09 on dist-test-slave-9k0h"
server_key: "ddcb4950d607b8246044c7c856fb1191"
server_key_iv: "c1bd5c97973cf2e7bc23169989469a09"
server_key_version: "encryptionkey@0"
I20260929 02:17:09.113404 11717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:09.133776 11717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:09.134564 11717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:09.134850 11717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:09.152755 11717 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:41587
I20260929 02:17:09.152761 11782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:41587 every 8 connection(s)
I20260929 02:17:09.154031 11717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:09.157505 11783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:09.163616 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 11717
I20260929 02:17:09.163811 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:09.164417 25442 external_mini_cluster.cc:1428] Setting key f7e1637afc2d920e4a6eede27cd13bbb
I20260929 02:17:09.168244 11783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90: Bootstrap starting.
I20260929 02:17:09.171562 11783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:09.172791 11783 log.cc:824] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:09.175881 11783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90: No bootstrap required, opened a new log
I20260929 02:17:09.177954 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:09.166310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.24.216.190:41587)
0929 02:17:09.167224 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:09.167234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:09.167274 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:17:09.168982 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:09.169004 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:09.169418 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:09.169798 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:09.170419 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:09.171712 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:09.173625 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:09.174475 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:09.174599 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:09.175770 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:09.175777 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:09.175789 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:09.176347 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:09.176427 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:09.176450 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:09.176533 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:09.176756 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 02:17:09.177061 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":328,"threads_started":1}
I20260929 02:17:09.183187 11783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a7ecec7bca4213ae2038acc2760f90" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41587 } }
I20260929 02:17:09.183517 11783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:09.183631 11783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a7ecec7bca4213ae2038acc2760f90, State: Initialized, Role: FOLLOWER
I20260929 02:17:09.184154 11783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a7ecec7bca4213ae2038acc2760f90" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41587 } }
I20260929 02:17:09.184410 11783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:09.184546 11783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:09.184719 11783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:09.186995 11783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a7ecec7bca4213ae2038acc2760f90" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41587 } }
I20260929 02:17:09.187493 11783 leader_election.cc:302] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99a7ecec7bca4213ae2038acc2760f90; no voters:
I20260929 02:17:09.188024 11783 leader_election.cc:288] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:09.188167 11788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:09.188807 11788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [term 1 LEADER]: Becoming Leader. State: Replica: 99a7ecec7bca4213ae2038acc2760f90, State: Running, Role: LEADER
I20260929 02:17:09.189373 11788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a7ecec7bca4213ae2038acc2760f90" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41587 } }
I20260929 02:17:09.190109 11783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:09.192561 11789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99a7ecec7bca4213ae2038acc2760f90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a7ecec7bca4213ae2038acc2760f90" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41587 } } }
I20260929 02:17:09.193257 11789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:09.192580 11790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99a7ecec7bca4213ae2038acc2760f90. Latest consensus state: current_term: 1 leader_uuid: "99a7ecec7bca4213ae2038acc2760f90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a7ecec7bca4213ae2038acc2760f90" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41587 } } }
I20260929 02:17:09.193449 11790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:09.198863 11795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:09.204293 11795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:09.213131 11795 catalog_manager.cc:1347] Generated new cluster ID: e95b513b90d843f0aa0245e484ee897f
I20260929 02:17:09.213243 11795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:09.226740 11795 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:09.227875 11795 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:09.240667 11795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90: Generated new TSK 0
I20260929 02:17:09.241428 11795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:09.247382 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41587
--builtin_ntp_servers=127.24.216.148:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:17:09.576743 11807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:09.577013 11807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:09.577178 11807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:09.584201 11807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:09.584286 11807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:09.584465 11807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:17:09.593338 11807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:09.594616 11807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:09.596324 11807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:09.608970 11812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:09.609010 11813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:09.609032 11815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:09.610291 11807 server_base.cc:1034] running on GCE node
I20260929 02:17:09.610870 11807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:09.611801 11807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:09.613030 11807 hybrid_clock.cc:648] HybridClock initialized: now 1790648229612992 us; error 46 us; skew 500 ppm
I20260929 02:17:09.613349 11807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:17:09.623301 11807 webserver.cc:460] Webserver started at http://127.24.216.129:41061/ using document root <none> and password file <none>
I20260929 02:17:09.624080 11807 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:09.624209 11807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:09.624518 11807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:09.627477 11807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "b4358e6490fe4e5584db8453e67f8bea"
format_stamp: "Formatted at 2026-09-29 02:17:09 on dist-test-slave-9k0h"
server_key: "744ae183f1cb2938c5e2bddaba774203"
server_key_iv: "a1a5449cc21c5126de178f6ba0cd0af8"
server_key_version: "encryptionkey@0"
I20260929 02:17:09.628242 11807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "b4358e6490fe4e5584db8453e67f8bea"
format_stamp: "Formatted at 2026-09-29 02:17:09 on dist-test-slave-9k0h"
server_key: "744ae183f1cb2938c5e2bddaba774203"
server_key_iv: "a1a5449cc21c5126de178f6ba0cd0af8"
server_key_version: "encryptionkey@0"
I20260929 02:17:09.633191 11807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:17:09.636415 11821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:09.637650 11807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:17:09.637812 11807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "b4358e6490fe4e5584db8453e67f8bea"
format_stamp: "Formatted at 2026-09-29 02:17:09 on dist-test-slave-9k0h"
server_key: "744ae183f1cb2938c5e2bddaba774203"
server_key_iv: "a1a5449cc21c5126de178f6ba0cd0af8"
server_key_version: "encryptionkey@0"
I20260929 02:17:09.638038 11807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:09.660840 11807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:09.661595 11807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:09.661854 11807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:09.662777 11807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:09.664084 11807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:09.664168 11807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:09.664244 11807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:09.664301 11807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:09.695881 11807 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:44955
I20260929 02:17:09.695966 11933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:44955 every 8 connection(s)
I20260929 02:17:09.697146 11807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:17:09.705880 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 11807
I20260929 02:17:09.706058 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:17:09.706605 25442 external_mini_cluster.cc:1428] Setting key 5e60cba9dbe10312efc897f0905d6829
I20260929 02:17:09.709677 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41587
--builtin_ntp_servers=127.24.216.148:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:17:09.709918 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:09.699669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:44987 (local address 127.24.216.190:41587)
0929 02:17:09.700125 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:09.700130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:09.701215 (+ 1085us) server_negotiation.cc:407] Connection header received
0929 02:17:09.703035 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:09.703041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:09.703127 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:09.703251 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:09.704255 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:09.704988 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:09.707126 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:09.707688 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:09.707827 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:09.709157 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:09.709161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:09.709165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:09.709181 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:09.709237 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:09.709249 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:09.709311 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:09.709551 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 02:17:09.709623 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":238,"threads_started":1}
I20260929 02:17:09.711336 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:09.700220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:41587 (local address 127.24.216.129:44987)
0929 02:17:09.701047 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:09.701079 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:09.702782 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:09.703461 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:09.703469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:09.703555 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:09.704092 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:09.704107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:09.705121 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:09.705125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:09.706991 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:09.707007 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:09.707881 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:09.707886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:09.708071 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:09.708825 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:17:09.708828 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:09.708875 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:17:09.708975 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:09.709341 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:09.709351 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:09.709360 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 02:17:09.709549 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 02:17:09.709967 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":253,"threads_started":1}
I20260929 02:17:09.712765 11934 heartbeater.cc:347] Connected to a master server at 127.24.216.190:41587
I20260929 02:17:09.713047 11934 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:09.713747 11934 heartbeater.cc:511] Master 127.24.216.190:41587 requested a full tablet report, sending...
I20260929 02:17:09.715675 11748 ts_manager.cc:194] Registered new tserver with Master: b4358e6490fe4e5584db8453e67f8bea (127.24.216.129:44955)
I20260929 02:17:09.717255 11748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.216.129:44987
W20260929 02:17:10.041301 11938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:10.041577 11938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:10.041677 11938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:10.048636 11938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:10.048723 11938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:10.048825 11938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:17:10.057266 11938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:10.058439 11938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:10.060122 11938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:10.072613 11944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:10.072592 11946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:10.072713 11943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:10.078780 11938 server_base.cc:1034] running on GCE node
I20260929 02:17:10.079452 11938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:10.080454 11938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:10.081674 11938 hybrid_clock.cc:648] HybridClock initialized: now 1790648230081615 us; error 49 us; skew 500 ppm
I20260929 02:17:10.081988 11938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:17:10.084264 11938 webserver.cc:460] Webserver started at http://127.24.216.130:41107/ using document root <none> and password file <none>
I20260929 02:17:10.084931 11938 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:10.085070 11938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:10.085377 11938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:10.088325 11938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "1403ad2253ff4694be1403398d254981"
format_stamp: "Formatted at 2026-09-29 02:17:10 on dist-test-slave-9k0h"
server_key: "fa4145fda60b8d0104e11ef0a120f41f"
server_key_iv: "39b1e7f36be61c2cd989cf1d99cc1b9a"
server_key_version: "encryptionkey@0"
I20260929 02:17:10.089085 11938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "1403ad2253ff4694be1403398d254981"
format_stamp: "Formatted at 2026-09-29 02:17:10 on dist-test-slave-9k0h"
server_key: "fa4145fda60b8d0104e11ef0a120f41f"
server_key_iv: "39b1e7f36be61c2cd989cf1d99cc1b9a"
server_key_version: "encryptionkey@0"
I20260929 02:17:10.093894 11938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260929 02:17:10.097226 11952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:10.098532 11938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:17:10.098695 11938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "1403ad2253ff4694be1403398d254981"
format_stamp: "Formatted at 2026-09-29 02:17:10 on dist-test-slave-9k0h"
server_key: "fa4145fda60b8d0104e11ef0a120f41f"
server_key_iv: "39b1e7f36be61c2cd989cf1d99cc1b9a"
server_key_version: "encryptionkey@0"
I20260929 02:17:10.098905 11938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:10.123940 11938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:10.124770 11938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:10.125073 11938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:10.126004 11938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:10.127475 11938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:10.127591 11938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:10.127719 11938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:10.127805 11938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:10.159955 11938 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:33973
I20260929 02:17:10.159992 12064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:33973 every 8 connection(s)
I20260929 02:17:10.161417 11938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:17:10.167227 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 11938
I20260929 02:17:10.167416 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:17:10.167970 25442 external_mini_cluster.cc:1428] Setting key d06b6fd78c21a72b2ecb34da8b0ade35
I20260929 02:17:10.170892 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41587
--builtin_ntp_servers=127.24.216.148:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:17:10.175446 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.163967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:48929 (local address 127.24.216.190:41587)
0929 02:17:10.164129 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:10.164134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:10.165540 (+ 1406us) server_negotiation.cc:407] Connection header received
0929 02:17:10.167604 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:10.167610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:10.167690 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:10.167796 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:10.169123 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.170051 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.172342 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.173045 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.173192 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.174575 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:10.174584 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:10.174591 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:10.174615 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:10.174654 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:10.174672 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:10.174766 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:10.174963 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 02:17:10.175072 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:17:10.176255 12067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.164540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:41587 (local address 127.24.216.130:48929)
0929 02:17:10.165388 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:10.165426 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:10.167269 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:10.168035 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:10.168047 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:10.168168 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:10.168903 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:10.168927 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.170222 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.170228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:10.172188 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:10.172202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.173268 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.173275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:10.173532 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.174290 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:17:10.174292 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:10.174329 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:17:10.174405 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:10.174769 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:10.174777 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:10.174782 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:17:10.174908 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 02:17:10.175218 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":266,"threads_started":1}
I20260929 02:17:10.177616 12065 heartbeater.cc:347] Connected to a master server at 127.24.216.190:41587
I20260929 02:17:10.177917 12065 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:10.178835 12065 heartbeater.cc:511] Master 127.24.216.190:41587 requested a full tablet report, sending...
I20260929 02:17:10.180359 11748 ts_manager.cc:194] Registered new tserver with Master: 1403ad2253ff4694be1403398d254981 (127.24.216.130:33973)
I20260929 02:17:10.181149 11748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.216.130:48929
W20260929 02:17:10.509398 12068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:10.509652 12068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:10.509796 12068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:10.517021 12068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:10.517122 12068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:10.517285 12068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:17:10.526098 12068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:41587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:10.527323 12068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:10.529042 12068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:10.541764 12074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:10.541760 12076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:10.542004 12073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:10.542860 12068 server_base.cc:1034] running on GCE node
I20260929 02:17:10.543484 12068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:10.544377 12068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:10.545595 12068 hybrid_clock.cc:648] HybridClock initialized: now 1790648230545544 us; error 54 us; skew 500 ppm
I20260929 02:17:10.545898 12068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:17:10.548153 12068 webserver.cc:460] Webserver started at http://127.24.216.131:45573/ using document root <none> and password file <none>
I20260929 02:17:10.548818 12068 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:10.548947 12068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:10.549258 12068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:10.552001 12068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "103edfbdc8d240d78d20671b4af8344a"
format_stamp: "Formatted at 2026-09-29 02:17:10 on dist-test-slave-9k0h"
server_key: "49e101988654a0e8e98a49412a982a7f"
server_key_iv: "90e39171d07908d536796001ac774c66"
server_key_version: "encryptionkey@0"
I20260929 02:17:10.552766 12068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "103edfbdc8d240d78d20671b4af8344a"
format_stamp: "Formatted at 2026-09-29 02:17:10 on dist-test-slave-9k0h"
server_key: "49e101988654a0e8e98a49412a982a7f"
server_key_iv: "90e39171d07908d536796001ac774c66"
server_key_version: "encryptionkey@0"
I20260929 02:17:10.557512 12068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260929 02:17:10.560858 12082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:10.562014 12068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 02:17:10.562237 12068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "103edfbdc8d240d78d20671b4af8344a"
format_stamp: "Formatted at 2026-09-29 02:17:10 on dist-test-slave-9k0h"
server_key: "49e101988654a0e8e98a49412a982a7f"
server_key_iv: "90e39171d07908d536796001ac774c66"
server_key_version: "encryptionkey@0"
I20260929 02:17:10.562445 12068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:10.573899 12068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:10.574754 12068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:10.575043 12068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:10.575996 12068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:10.577358 12068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:10.577473 12068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:10.577605 12068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:10.577696 12068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:10.610296 12068 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:36203
I20260929 02:17:10.610334 12194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:36203 every 8 connection(s)
I20260929 02:17:10.611581 12068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:17:10.618338 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 12068
I20260929 02:17:10.618501 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:17:10.618908 25442 external_mini_cluster.cc:1428] Setting key 63cb2bb2ac7e8ac2c3a0636b00b20055
I20260929 02:17:10.623759 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.613933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:52109 (local address 127.24.216.190:41587)
0929 02:17:10.614120 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:10.614127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:10.615451 (+ 1324us) server_negotiation.cc:407] Connection header received
0929 02:17:10.617344 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:10.617350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:10.617443 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:10.617569 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:10.618775 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.619568 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.621082 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.621506 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.621641 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.622922 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:10.622927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:10.622930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:10.622947 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:10.622977 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:10.622988 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:10.623084 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:10.623313 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 02:17:10.623406 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:17:10.624647 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.614460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:41587 (local address 127.24.216.131:52109)
0929 02:17:10.615290 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:10.615326 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:10.617076 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:10.617784 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:10.617793 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:10.617880 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:10.618612 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:10.618633 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.619720 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.619726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:10.620931 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:10.620942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.621723 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.621730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:10.621948 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.622612 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:17:10.622615 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:10.622657 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:17:10.622727 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:10.623079 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:10.623089 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:10.623099 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:17:10.623294 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 02:17:10.623646 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":250,"threads_started":1}
I20260929 02:17:10.625872 12195 heartbeater.cc:347] Connected to a master server at 127.24.216.190:41587
I20260929 02:17:10.626107 12195 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:10.626811 12195 heartbeater.cc:511] Master 127.24.216.190:41587 requested a full tablet report, sending...
I20260929 02:17:10.628083 11748 ts_manager.cc:194] Registered new tserver with Master: 103edfbdc8d240d78d20671b4af8344a (127.24.216.131:36203)
I20260929 02:17:10.628868 11748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.216.131:52109
I20260929 02:17:10.633365 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:17:10.643524 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.637659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.24.216.190:41587)
0929 02:17:10.637861 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:10.637868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:10.638275 (+ 407us) server_negotiation.cc:407] Connection header received
0929 02:17:10.638472 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:10.638476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:10.638541 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:10.638639 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:10.639094 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.639767 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.641106 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.641518 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.641651 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.642831 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:10.642835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:10.642838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:10.642855 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:10.642884 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:10.642895 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:10.643039 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:10.643222 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 02:17:10.643274 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:17:10.647078 11748 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:17:10.653719 11748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:17:10.678339 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.664984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.24.216.129:44955)
0929 02:17:10.665631 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:10.665639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:10.667254 (+ 1615us) server_negotiation.cc:407] Connection header received
0929 02:17:10.667355 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:10.667364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:10.667626 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:10.667859 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:10.669493 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.671506 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.675138 (+ 3632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.676970 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.677128 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.677312 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:10.677806 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 02:17:10.677969 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":512,"threads_started":1}
I20260929 02:17:10.678325 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.664776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:44955 (local address 127.0.0.1:41448)
0929 02:17:10.666810 (+ 2034us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:10.666856 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:10.667040 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:10.668460 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:10.668471 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:10.668522 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:10.669090 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:10.669106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.671719 (+ 2613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.671727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:10.674884 (+ 3157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:10.674904 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.677425 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.677430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:10.677580 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.677617 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:17:10.677767 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 02:17:10.677946 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1786,"thread_start_us":12997,"threads_started":1}
I20260929 02:17:10.682431 11869 tablet_service.cc:1467] Processing CreateTablet for tablet 7fedf87d0592468384e9582f1bd568ff (DEFAULT_TABLE table=test-table [id=01387f90636e4cc2b4a30dd8644f60f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:10.683995 11869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fedf87d0592468384e9582f1bd568ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:10.685226 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.665965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:36203 (local address 127.0.0.1:34394)
0929 02:17:10.677741 (+ 11776us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:10.677758 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:10.677875 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:10.678574 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:10.678581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:10.678611 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:10.678993 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:10.679001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.680908 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.680915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:10.682364 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:10.682376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.684396 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.684402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:10.684588 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.684620 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:17:10.684742 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 02:17:10.684867 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11613,"mutex_wait_us":1,"thread_start_us":1025,"threads_started":1}
I20260929 02:17:10.685326 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.666657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.24.216.131:36203)
0929 02:17:10.676952 (+ 10295us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:10.676960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:10.677999 (+ 1039us) server_negotiation.cc:407] Connection header received
0929 02:17:10.678103 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:10.678113 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:10.678366 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:10.679132 (+ 766us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:10.679418 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.680766 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.682541 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.683877 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.684060 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.684254 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:10.684803 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 02:17:10.684945 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10115,"thread_start_us":10303,"threads_started":1}
I20260929 02:17:10.685360 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.665360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.24.216.130:33973)
0929 02:17:10.666126 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:10.666134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:10.676186 (+ 10052us) server_negotiation.cc:407] Connection header received
0929 02:17:10.677120 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:10.677135 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:10.677409 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:10.677587 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:10.678874 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.680345 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.682815 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.684245 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.684419 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.684612 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:10.684854 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 02:17:10.685032 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":470,"threads_started":1}
I20260929 02:17:10.685956 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.665034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:33973 (local address 127.0.0.1:51118)
0929 02:17:10.676053 (+ 11019us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:10.676071 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:10.676771 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:10.677999 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:10.678003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:10.678026 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:10.678414 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:10.678664 (+ 250us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.680519 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.680527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:10.682622 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:10.682636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.684449 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.684455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:10.684649 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.684683 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 02:17:10.685535 (+ 852us) client_negotiation.cc:240] Negotiation successful
0929 02:17:10.685669 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10878,"thread_start_us":10755,"threads_started":1}
I20260929 02:17:10.689208 12130 tablet_service.cc:1467] Processing CreateTablet for tablet 7fedf87d0592468384e9582f1bd568ff (DEFAULT_TABLE table=test-table [id=01387f90636e4cc2b4a30dd8644f60f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:10.689285 12000 tablet_service.cc:1467] Processing CreateTablet for tablet 7fedf87d0592468384e9582f1bd568ff (DEFAULT_TABLE table=test-table [id=01387f90636e4cc2b4a30dd8644f60f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:10.690591 12130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fedf87d0592468384e9582f1bd568ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:10.690616 12000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fedf87d0592468384e9582f1bd568ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:10.706799 12213 tablet_bootstrap.cc:492] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea: Bootstrap starting.
I20260929 02:17:10.709657 12214 tablet_bootstrap.cc:492] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a: Bootstrap starting.
I20260929 02:17:10.711601 12213 tablet_bootstrap.cc:654] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:10.713156 12215 tablet_bootstrap.cc:492] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981: Bootstrap starting.
I20260929 02:17:10.713301 12213 log.cc:824] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:10.714340 12214 tablet_bootstrap.cc:654] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:10.715553 12214 log.cc:824] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:10.716054 12213 tablet_bootstrap.cc:492] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea: No bootstrap required, opened a new log
I20260929 02:17:10.716398 12213 ts_tablet_manager.cc:1397] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20260929 02:17:10.717512 12215 tablet_bootstrap.cc:654] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:10.718753 12214 tablet_bootstrap.cc:492] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a: No bootstrap required, opened a new log
I20260929 02:17:10.718888 12215 log.cc:824] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:10.719008 12214 ts_tablet_manager.cc:1397] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260929 02:17:10.720621 11934 heartbeater.cc:503] Master 127.24.216.190:41587 was elected leader, sending a full tablet report...
I20260929 02:17:10.721993 12215 tablet_bootstrap.cc:492] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981: No bootstrap required, opened a new log
I20260929 02:17:10.722326 12215 ts_tablet_manager.cc:1397] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:17:10.725816 12213 raft_consensus.cc:348] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4358e6490fe4e5584db8453e67f8bea" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 44955 } } peers { permanent_uuid: "1403ad2253ff4694be1403398d254981" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33973 } } peers { permanent_uuid: "103edfbdc8d240d78d20671b4af8344a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36203 } }
I20260929 02:17:10.726178 12213 raft_consensus.cc:374] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:10.726300 12213 raft_consensus.cc:729] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4358e6490fe4e5584db8453e67f8bea, State: Initialized, Role: FOLLOWER
I20260929 02:17:10.726893 12213 consensus_queue.cc:260] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4358e6490fe4e5584db8453e67f8bea" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 44955 } } peers { permanent_uuid: "1403ad2253ff4694be1403398d254981" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33973 } } peers { permanent_uuid: "103edfbdc8d240d78d20671b4af8344a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36203 } }
I20260929 02:17:10.728596 12213 ts_tablet_manager.cc:1428] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260929 02:17:10.729135 12214 raft_consensus.cc:348] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4358e6490fe4e5584db8453e67f8bea" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 44955 } } peers { permanent_uuid: "1403ad2253ff4694be1403398d254981" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33973 } } peers { permanent_uuid: "103edfbdc8d240d78d20671b4af8344a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36203 } }
I20260929 02:17:10.729550 12214 raft_consensus.cc:374] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:10.729650 12214 raft_consensus.cc:729] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103edfbdc8d240d78d20671b4af8344a, State: Initialized, Role: FOLLOWER
I20260929 02:17:10.730234 12214 consensus_queue.cc:260] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4358e6490fe4e5584db8453e67f8bea" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 44955 } } peers { permanent_uuid: "1403ad2253ff4694be1403398d254981" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33973 } } peers { permanent_uuid: "103edfbdc8d240d78d20671b4af8344a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36203 } }
I20260929 02:17:10.731324 12195 heartbeater.cc:503] Master 127.24.216.190:41587 was elected leader, sending a full tablet report...
I20260929 02:17:10.731837 12214 ts_tablet_manager.cc:1428] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 02:17:10.733670 12215 raft_consensus.cc:348] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4358e6490fe4e5584db8453e67f8bea" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 44955 } } peers { permanent_uuid: "1403ad2253ff4694be1403398d254981" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33973 } } peers { permanent_uuid: "103edfbdc8d240d78d20671b4af8344a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36203 } }
I20260929 02:17:10.734071 12215 raft_consensus.cc:374] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:10.734206 12215 raft_consensus.cc:729] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1403ad2253ff4694be1403398d254981, State: Initialized, Role: FOLLOWER
I20260929 02:17:10.734788 12215 consensus_queue.cc:260] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4358e6490fe4e5584db8453e67f8bea" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 44955 } } peers { permanent_uuid: "1403ad2253ff4694be1403398d254981" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33973 } } peers { permanent_uuid: "103edfbdc8d240d78d20671b4af8344a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36203 } }
I20260929 02:17:10.735798 12065 heartbeater.cc:503] Master 127.24.216.190:41587 was elected leader, sending a full tablet report...
I20260929 02:17:10.736306 12215 ts_tablet_manager.cc:1428] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260929 02:17:10.863247 12196 tablet.cc:2369] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:17:10.894766 12219 raft_consensus.cc:482] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:17:10.895108 12219 raft_consensus.cc:504] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4358e6490fe4e5584db8453e67f8bea" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 44955 } } peers { permanent_uuid: "1403ad2253ff4694be1403398d254981" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33973 } } peers { permanent_uuid: "103edfbdc8d240d78d20671b4af8344a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36203 } }
I20260929 02:17:10.896821 12219 leader_election.cc:288] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1403ad2253ff4694be1403398d254981 (127.24.216.130:33973), 103edfbdc8d240d78d20671b4af8344a (127.24.216.131:36203)
I20260929 02:17:10.904189 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.897386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:57365 (local address 127.24.216.130:33973)
0929 02:17:10.897639 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:10.897648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:10.898396 (+ 748us) server_negotiation.cc:407] Connection header received
0929 02:17:10.898597 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:10.898604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:10.898683 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:10.898777 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:10.899444 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.900216 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.902391 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.903241 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.903355 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.903402 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:10.903826 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 02:17:10.903944 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 02:17:10.904198 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.897239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:55517 (local address 127.24.216.131:36203)
0929 02:17:10.897491 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:10.897500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:10.898612 (+ 1112us) server_negotiation.cc:407] Connection header received
0929 02:17:10.898794 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:10.898799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:10.898878 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:10.898981 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:10.899600 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.900347 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.902433 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.903379 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.903534 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.903599 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:10.903883 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 02:17:10.903992 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:17:10.904307 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.897004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:33973 (local address 127.24.216.129:57365)
0929 02:17:10.898259 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:10.898284 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:10.898441 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:10.898800 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:10.898805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:10.898838 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:10.899295 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:10.899307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.900344 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.900350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:10.902199 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:10.902219 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.903409 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.903418 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:10.903612 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.903671 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 02:17:10.903760 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 02:17:10.903911 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"mutex_wait_us":93,"thread_start_us":817,"threads_started":1}
I20260929 02:17:10.904363 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.897177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:36203 (local address 127.24.216.129:55517)
0929 02:17:10.898473 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:10.898515 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:10.898647 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:10.899006 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:10.899009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:10.899038 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:10.899397 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:10.899408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.900472 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.900475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:10.902240 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:10.902252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.903568 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.903572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:10.903687 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.903709 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:17:10.903827 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 02:17:10.903943 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":40,"thread_start_us":909,"threads_started":1}
I20260929 02:17:10.905385 12020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fedf87d0592468384e9582f1bd568ff" candidate_uuid: "b4358e6490fe4e5584db8453e67f8bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1403ad2253ff4694be1403398d254981" is_pre_election: true
I20260929 02:17:10.905613 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fedf87d0592468384e9582f1bd568ff" candidate_uuid: "b4358e6490fe4e5584db8453e67f8bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103edfbdc8d240d78d20671b4af8344a" is_pre_election: true
I20260929 02:17:10.905902 12020 raft_consensus.cc:2454] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4358e6490fe4e5584db8453e67f8bea in term 0.
I20260929 02:17:10.906133 12150 raft_consensus.cc:2454] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4358e6490fe4e5584db8453e67f8bea in term 0.
I20260929 02:17:10.906682 11822 leader_election.cc:302] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1403ad2253ff4694be1403398d254981, b4358e6490fe4e5584db8453e67f8bea; no voters:
I20260929 02:17:10.907119 12219 raft_consensus.cc:2781] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:17:10.907266 12219 raft_consensus.cc:482] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:17:10.907372 12219 raft_consensus.cc:3037] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:10.909432 12219 raft_consensus.cc:504] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4358e6490fe4e5584db8453e67f8bea" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 44955 } } peers { permanent_uuid: "1403ad2253ff4694be1403398d254981" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33973 } } peers { permanent_uuid: "103edfbdc8d240d78d20671b4af8344a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36203 } }
I20260929 02:17:10.910049 12219 leader_election.cc:288] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [CANDIDATE]: Term 1 election: Requested vote from peers 1403ad2253ff4694be1403398d254981 (127.24.216.130:33973), 103edfbdc8d240d78d20671b4af8344a (127.24.216.131:36203)
I20260929 02:17:10.910511 12020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fedf87d0592468384e9582f1bd568ff" candidate_uuid: "b4358e6490fe4e5584db8453e67f8bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1403ad2253ff4694be1403398d254981"
I20260929 02:17:10.910583 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fedf87d0592468384e9582f1bd568ff" candidate_uuid: "b4358e6490fe4e5584db8453e67f8bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103edfbdc8d240d78d20671b4af8344a"
I20260929 02:17:10.910738 12020 raft_consensus.cc:3037] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:10.910758 12150 raft_consensus.cc:3037] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a [term 0 FOLLOWER]: Advancing to term 1
W20260929 02:17:10.913476 12066 tablet.cc:2369] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:17:10.913978 12150 raft_consensus.cc:2454] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4358e6490fe4e5584db8453e67f8bea in term 1.
I20260929 02:17:10.914001 12020 raft_consensus.cc:2454] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4358e6490fe4e5584db8453e67f8bea in term 1.
I20260929 02:17:10.914407 11823 leader_election.cc:302] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 103edfbdc8d240d78d20671b4af8344a, b4358e6490fe4e5584db8453e67f8bea; no voters:
I20260929 02:17:10.914713 12219 raft_consensus.cc:2781] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:10.915132 12219 raft_consensus.cc:686] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [term 1 LEADER]: Becoming Leader. State: Replica: b4358e6490fe4e5584db8453e67f8bea, State: Running, Role: LEADER
I20260929 02:17:10.915613 12219 consensus_queue.cc:237] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4358e6490fe4e5584db8453e67f8bea" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 44955 } } peers { permanent_uuid: "1403ad2253ff4694be1403398d254981" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33973 } } peers { permanent_uuid: "103edfbdc8d240d78d20671b4af8344a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36203 } }
I20260929 02:17:10.919910 11748 catalog_manager.cc:5495] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea reported cstate change: term changed from 0 to 1, leader changed from <none> to b4358e6490fe4e5584db8453e67f8bea (127.24.216.129). New cstate: current_term: 1 leader_uuid: "b4358e6490fe4e5584db8453e67f8bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4358e6490fe4e5584db8453e67f8bea" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 44955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1403ad2253ff4694be1403398d254981" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "103edfbdc8d240d78d20671b4af8344a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36203 } health_report { overall_health: UNKNOWN } } }
W20260929 02:17:10.949435 11935 tablet.cc:2369] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:17:10.982935 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:10.977541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.24.216.129:44955)
0929 02:17:10.977734 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:10.977739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:10.977759 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:17:10.977857 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:10.977860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:10.977946 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:10.978067 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:10.978477 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.979179 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.980637 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:10.981039 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:10.981135 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:10.981644 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:10.982141 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:10.982425 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:10.982578 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 02:17:10.982639 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:17:10.991269 12150 raft_consensus.cc:1261] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a [term 1 FOLLOWER]: Refusing update from remote peer b4358e6490fe4e5584db8453e67f8bea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:17:10.991292 12020 raft_consensus.cc:1261] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981 [term 1 FOLLOWER]: Refusing update from remote peer b4358e6490fe4e5584db8453e67f8bea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:17:10.992295 12219 consensus_queue.cc:1035] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1403ad2253ff4694be1403398d254981" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 33973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:17:10.992590 12224 consensus_queue.cc:1035] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "103edfbdc8d240d78d20671b4af8344a" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 36203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:17:11.017520 11748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:17:11.017824 11748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:17:11.021530 11748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99a7ecec7bca4213ae2038acc2760f90: Sending DeleteTablet for 3 replicas of tablet 7fedf87d0592468384e9582f1bd568ff
I20260929 02:17:11.022646 11869 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fedf87d0592468384e9582f1bd568ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:11 UTC) from {username='slave'} at 127.0.0.1:41448
I20260929 02:17:11.022804 12000 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fedf87d0592468384e9582f1bd568ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:11 UTC) from {username='slave'} at 127.0.0.1:51118
I20260929 02:17:11.022969 12130 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fedf87d0592468384e9582f1bd568ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:11 UTC) from {username='slave'} at 127.0.0.1:34394
I20260929 02:17:11.024711 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 11807
I20260929 02:17:11.039770 12238 tablet_replica.cc:330] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981: stopping tablet replica
I20260929 02:17:11.039973 12237 tablet_replica.cc:330] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea: stopping tablet replica
I20260929 02:17:11.040407 12238 raft_consensus.cc:2229] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:17:11.040472 12237 raft_consensus.cc:2229] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:17:11.041080 12238 raft_consensus.cc:2258] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:11.041440 12237 raft_consensus.cc:2258] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:11.044044 12237 ts_tablet_manager.cc:1905] T 7fedf87d0592468384e9582f1bd568ff P b4358e6490fe4e5584db8453e67f8bea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:11.048643 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:11.025963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.24.216.129:44955)
0929 02:17:11.026140 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:11.026166 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:11.031338 (+ 5172us) server_negotiation.cc:407] Connection header received
0929 02:17:11.032313 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:11.032326 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:11.032482 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:11.032573 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:11.033917 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:11.035578 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:11.041427 (+ 5849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:11.042393 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:11.042531 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:11.046327 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:11.046342 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:11.046364 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:11.047414 (+ 1050us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:11.047491 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:11.047521 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:11.047618 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:11.047937 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 02:17:11.048081 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:17:11.050359 11929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:11.051195 12238 ts_tablet_manager.cc:1905] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:11.052222 12239 tablet_replica.cc:330] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a: stopping tablet replica
I20260929 02:17:11.052765 12239 raft_consensus.cc:2229] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:17:11.053362 12239 raft_consensus.cc:2258] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:11.055841 12239 ts_tablet_manager.cc:1905] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:11.059055 12238 ts_tablet_manager.cc:1918] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:11.059224 12238 log.cc:1196] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/7fedf87d0592468384e9582f1bd568ff
I20260929 02:17:11.059798 12238 ts_tablet_manager.cc:1939] T 7fedf87d0592468384e9582f1bd568ff P 1403ad2253ff4694be1403398d254981: Deleting consensus metadata
I20260929 02:17:11.061759 11732 catalog_manager.cc:4841] TS 1403ad2253ff4694be1403398d254981 (127.24.216.130:33973): tablet 7fedf87d0592468384e9582f1bd568ff (table test-table [id=01387f90636e4cc2b4a30dd8644f60f3]) successfully deleted
I20260929 02:17:11.062815 12239 ts_tablet_manager.cc:1918] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:11.062965 12239 log.cc:1196] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/7fedf87d0592468384e9582f1bd568ff
I20260929 02:17:11.063372 12239 ts_tablet_manager.cc:1939] T 7fedf87d0592468384e9582f1bd568ff P 103edfbdc8d240d78d20671b4af8344a: Deleting consensus metadata
I20260929 02:17:11.064706 11733 catalog_manager.cc:4841] TS 103edfbdc8d240d78d20671b4af8344a (127.24.216.131:36203): tablet 7fedf87d0592468384e9582f1bd568ff (table test-table [id=01387f90636e4cc2b4a30dd8644f60f3]) successfully deleted
I20260929 02:17:11.149541 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 11807
W20260929 02:17:11.170275 11734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.216.129:44955 (error 108)
I20260929 02:17:11.170463 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 11938
I20260929 02:17:11.172295 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:11.171795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:44955 (local address 127.0.0.1:41482)
0929 02:17:11.171989 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:11.172116 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:44955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 02:17:11.172555 11734 catalog_manager.cc:4584] TS b4358e6490fe4e5584db8453e67f8bea (127.24.216.129:44955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7fedf87d0592468384e9582f1bd568ff: Network error: Client connection negotiation failed: client connection to 127.24.216.129:44955: connect: Connection refused (error 111)
I20260929 02:17:11.178318 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:11.171458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51134 (local address 127.24.216.130:33973)
0929 02:17:11.171651 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:11.171657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:11.171678 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:17:11.171746 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:11.171751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:11.171853 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:11.171955 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:11.172502 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:11.173396 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:11.175236 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:11.175666 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:11.175762 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:11.176870 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:11.176882 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:11.176896 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:11.177535 (+ 639us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:11.177590 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:11.177609 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:11.177697 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:11.177859 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 02:17:11.177944 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:17:11.179018 12060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:11.226223 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:11.225685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:44955 (local address 127.0.0.1:41498)
0929 02:17:11.225899 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:11.226011 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:44955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 02:17:11.227237 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:11.226885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:44955 (local address 127.0.0.1:41512)
0929 02:17:11.227036 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:11.227114 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:44955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 02:17:11.272703 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 11938
I20260929 02:17:11.287636 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:11.287181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:44955 (local address 127.0.0.1:41522)
0929 02:17:11.287382 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:11.287476 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:44955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 02:17:11.288648 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:11.288313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:44955 (local address 127.0.0.1:41530)
0929 02:17:11.288462 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:11.288534 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:44955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 02:17:11.291754 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 12068
I20260929 02:17:11.298265 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:11.292579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.24.216.131:36203)
0929 02:17:11.292754 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:11.292759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:11.292838 (+ 79us) server_negotiation.cc:407] Connection header received
0929 02:17:11.292980 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:11.292984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:11.293095 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:11.293183 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:11.293525 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:11.294313 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:11.295555 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:11.295937 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:11.296028 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:11.296938 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:11.296946 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:11.296962 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:11.297523 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:11.297569 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:11.297594 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:11.297678 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:11.297784 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 02:17:11.297861 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:17:11.298875 12190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:11.368913 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:11.368421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:44955 (local address 127.0.0.1:41540)
0929 02:17:11.368654 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:11.368750 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:44955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 02:17:11.370034 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:11.369628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:44955 (local address 127.0.0.1:41556)
0929 02:17:11.369867 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:11.369934 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.129:44955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 02:17:11.386734 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 12068
I20260929 02:17:11.404358 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 11717
I20260929 02:17:11.405419 11778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:11.513486 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 11717
2026-09-29T02:17:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2848 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:11 dist-test-slave-9k0h krb5kdc[12248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:11 dist-test-slave-9k0h krb5kdc[12248](info): set up 2 sockets
Sep 29 02:17:11 dist-test-slave-9k0h krb5kdc[12248](info): commencing operation
krb5kdc: starting...
W20260929 02:17:12.614445 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:12 dist-test-slave-9k0h krb5kdc[12248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:17:12.903484 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:37475
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:37475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:13.231444 12261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:13.231763 12261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:13.231940 12261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:13.239446 12261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:13.239570 12261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:13.239655 12261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:13.239753 12261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:13.239840 12261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:13.249540 12261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:37475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:37475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:13.250927 12261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:13.252220 12261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:13.264042 12267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:13.264012 12266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:13.264117 12269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:13.264652 12261 server_base.cc:1034] running on GCE node
I20260929 02:17:13.265393 12261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:13.266775 12261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:13.267969 12261 hybrid_clock.cc:648] HybridClock initialized: now 1790648233267935 us; error 23 us; skew 500 ppm
I20260929 02:17:13.268313 12261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:13 dist-test-slave-9k0h krb5kdc[12248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648233, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:13.271281 12261 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:17:13.272684 12261 webserver.cc:460] Webserver started at http://127.24.216.190:44801/ using document root <none> and password file <none>
I20260929 02:17:13.273295 12261 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:13.273391 12261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:13.273629 12261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:13.276492 12261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "448a0a64f2ac4e848118c8f81d5a86a0"
format_stamp: "Formatted at 2026-09-29 02:17:13 on dist-test-slave-9k0h"
server_key: "dbf9c2b3c3a41cc8a3dc49bf2fa5a19d"
server_key_iv: "66e52828b835edc514e8f89ffa6cd85d"
server_key_version: "encryptionkey@0"
I20260929 02:17:13.277163 12261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "448a0a64f2ac4e848118c8f81d5a86a0"
format_stamp: "Formatted at 2026-09-29 02:17:13 on dist-test-slave-9k0h"
server_key: "dbf9c2b3c3a41cc8a3dc49bf2fa5a19d"
server_key_iv: "66e52828b835edc514e8f89ffa6cd85d"
server_key_version: "encryptionkey@0"
I20260929 02:17:13.282021 12261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260929 02:17:13.285318 12276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:13.286675 12261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:17:13.286845 12261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "448a0a64f2ac4e848118c8f81d5a86a0"
format_stamp: "Formatted at 2026-09-29 02:17:13 on dist-test-slave-9k0h"
server_key: "dbf9c2b3c3a41cc8a3dc49bf2fa5a19d"
server_key_iv: "66e52828b835edc514e8f89ffa6cd85d"
server_key_version: "encryptionkey@0"
I20260929 02:17:13.287020 12261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:13.303377 12261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:13.307098 12261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:13.307376 12261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:13.325209 12261 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:37475
I20260929 02:17:13.325233 12327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:37475 every 8 connection(s)
I20260929 02:17:13.326632 12261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:13.329509 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 12261
I20260929 02:17:13.329702 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:13.330291 25442 external_mini_cluster.cc:1428] Setting key f1d3e899e98e36e289f66395058f8bb7
I20260929 02:17:13.330623 12328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:13.344808 12328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0: Bootstrap starting.
Sep 29 02:17:13 dist-test-slave-9k0h krb5kdc[12248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:17:13.348213 12328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:13.349257 12328 log.cc:824] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:13.352485 12328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0: No bootstrap required, opened a new log
I20260929 02:17:13.353390 12331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:13.332064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47924 (local address 127.24.216.190:37475)
0929 02:17:13.332805 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:13.332817 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:13.332855 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:17:13.335073 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:13.335093 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:13.335462 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:13.335786 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:13.337527 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:13.338852 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:13.340653 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:13.341339 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:13.341471 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:13.346283 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:13.346291 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:13.346304 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:13.346338 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:13.349285 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:13.350199 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:13.350209 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:13.350219 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:13.350432 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:13.350893 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:13.350896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:13.350898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:13.351355 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:13.351571 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:13.352092 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 02:17:13.352375 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":350,"threads_started":1}
I20260929 02:17:13.360054 12328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448a0a64f2ac4e848118c8f81d5a86a0" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37475 } }
I20260929 02:17:13.360469 12328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:13.360572 12328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448a0a64f2ac4e848118c8f81d5a86a0, State: Initialized, Role: FOLLOWER
I20260929 02:17:13.361090 12328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448a0a64f2ac4e848118c8f81d5a86a0" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37475 } }
I20260929 02:17:13.361325 12328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:13.361472 12328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:13.361639 12328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:13.363893 12328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448a0a64f2ac4e848118c8f81d5a86a0" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37475 } }
I20260929 02:17:13.364365 12328 leader_election.cc:302] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 448a0a64f2ac4e848118c8f81d5a86a0; no voters:
I20260929 02:17:13.364905 12328 leader_election.cc:288] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:13.365069 12333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:13.365748 12333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [term 1 LEADER]: Becoming Leader. State: Replica: 448a0a64f2ac4e848118c8f81d5a86a0, State: Running, Role: LEADER
I20260929 02:17:13.366282 12333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448a0a64f2ac4e848118c8f81d5a86a0" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37475 } }
I20260929 02:17:13.367007 12328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:13.369004 12334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "448a0a64f2ac4e848118c8f81d5a86a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448a0a64f2ac4e848118c8f81d5a86a0" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37475 } } }
I20260929 02:17:13.369045 12335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 448a0a64f2ac4e848118c8f81d5a86a0. Latest consensus state: current_term: 1 leader_uuid: "448a0a64f2ac4e848118c8f81d5a86a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448a0a64f2ac4e848118c8f81d5a86a0" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37475 } } }
I20260929 02:17:13.369397 12334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:13.369552 12335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:13.376917 12338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:13.381080 12338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:13.389202 12338 catalog_manager.cc:1347] Generated new cluster ID: c06bfbca23ef4456a4da4262374b2440
I20260929 02:17:13.389302 12338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:13.403723 12338 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:13.404923 12338 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:13.417789 12338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0: Generated new TSK 0
I20260929 02:17:13.418614 12338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.216.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
Entry for principal kudu/127.24.216.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab.
I20260929 02:17:13.485749 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37475
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:13.809870 12354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:13.810201 12354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:13.810385 12354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:13.817530 12354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:13.817682 12354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:13.817857 12354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:17:13.828161 12354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.129.keytab
--principal=kudu/127.24.216.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:13.829509 12354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:13.830911 12354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:13.844194 12359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:13.844242 12362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:13.844278 12360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:13.845312 12354 server_base.cc:1034] running on GCE node
I20260929 02:17:13.845887 12354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:13.846813 12354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:13.848004 12354 hybrid_clock.cc:648] HybridClock initialized: now 1790648233847964 us; error 30 us; skew 500 ppm
I20260929 02:17:13.848307 12354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:13 dist-test-slave-9k0h krb5kdc[12248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648233, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:13.851282 12354 init.cc:377] Logged in from keytab as kudu/127.24.216.129@KRBTEST.COM (short username kudu)
I20260929 02:17:13.852629 12354 webserver.cc:460] Webserver started at http://127.24.216.129:35863/ using document root <none> and password file <none>
I20260929 02:17:13.853310 12354 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:13.853468 12354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:13.853773 12354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:13.856825 12354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "5990271b7a6a4d75acf823ebedefd33b"
format_stamp: "Formatted at 2026-09-29 02:17:13 on dist-test-slave-9k0h"
server_key: "eb3880430bf9122c0156b2ea3d9d73a0"
server_key_iv: "b7f35640ff6f6bcb07f5e62b26230b93"
server_key_version: "encryptionkey@0"
I20260929 02:17:13.857558 12354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "5990271b7a6a4d75acf823ebedefd33b"
format_stamp: "Formatted at 2026-09-29 02:17:13 on dist-test-slave-9k0h"
server_key: "eb3880430bf9122c0156b2ea3d9d73a0"
server_key_iv: "b7f35640ff6f6bcb07f5e62b26230b93"
server_key_version: "encryptionkey@0"
I20260929 02:17:13.862871 12354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:17:13.866063 12369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:13.867381 12354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:17:13.867543 12354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "5990271b7a6a4d75acf823ebedefd33b"
format_stamp: "Formatted at 2026-09-29 02:17:13 on dist-test-slave-9k0h"
server_key: "eb3880430bf9122c0156b2ea3d9d73a0"
server_key_iv: "b7f35640ff6f6bcb07f5e62b26230b93"
server_key_version: "encryptionkey@0"
I20260929 02:17:13.867738 12354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:13.884406 12354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:13.888314 12354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:13.888597 12354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:13.889508 12354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:13.890877 12354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:13.890973 12354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:13.891121 12354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:13.891191 12354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:13.924402 12354 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:41777
I20260929 02:17:13.924451 12481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:41777 every 8 connection(s)
I20260929 02:17:13.925683 12354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:17:13.933154 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 12354
I20260929 02:17:13.933358 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:17:13.933789 25442 external_mini_cluster.cc:1428] Setting key c112aa6921d338062b7c98c017b7598a
Sep 29 02:17:13 dist-test-slave-9k0h krb5kdc[12248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648233, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.129@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:17:13.947320 12484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:13.928269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:58093 (local address 127.24.216.190:37475)
0929 02:17:13.928772 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:13.928779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:13.929683 (+ 904us) server_negotiation.cc:407] Connection header received
0929 02:17:13.931506 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:13.931510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:13.931578 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:13.931684 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:13.933240 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:13.934136 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:13.936064 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:13.936461 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:13.936557 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:13.940746 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:13.940755 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:13.940762 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:13.940805 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:13.944265 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:13.944972 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:13.944980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:13.944986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:13.945130 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:13.945528 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:13.945536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:13.945542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:13.945920 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:13.946069 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:13.946832 (+ 763us) server_negotiation.cc:299] Negotiation successful
0929 02:17:13.947058 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":217,"threads_started":1}
I20260929 02:17:13.948648 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:13.928849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:37475 (local address 127.24.216.129:58093)
0929 02:17:13.929524 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:13.929561 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:13.931260 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:13.931894 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:13.931901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:13.932370 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:13.933054 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:13.933080 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:13.934330 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:13.934335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:13.935927 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:13.935942 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:13.936682 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:13.936691 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:13.937025 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:13.937741 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:13.937775 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:13.940519 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:13.944414 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:13.944421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:13.944434 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:13.944856 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:13.945238 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:13.945241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:13.945243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:13.945422 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:13.946035 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:13.946040 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:13.946421 (+ 381us) client_negotiation.cc:769] Sending connection context
0929 02:17:13.946678 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 02:17:13.947161 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":271,"threads_started":1}
I20260929 02:17:13.950196 12482 heartbeater.cc:347] Connected to a master server at 127.24.216.190:37475
I20260929 02:17:13.950492 12482 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:13.951203 12482 heartbeater.cc:511] Master 127.24.216.190:37475 requested a full tablet report, sending...
I20260929 02:17:13.953186 12293 ts_manager.cc:194] Registered new tserver with Master: 5990271b7a6a4d75acf823ebedefd33b (127.24.216.129:41777)
I20260929 02:17:13.954640 12293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.129@KRBTEST.COM'} at 127.24.216.129:58093
WARNING: no policy specified for kudu/127.24.216.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
Entry for principal kudu/127.24.216.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab.
I20260929 02:17:13.995677 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37475
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:14.323226 12488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:14.323527 12488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:14.323689 12488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:14.334506 12488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:14.334636 12488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:14.334828 12488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:17:14.349181 12488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.130.keytab
--principal=kudu/127.24.216.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:14.350373 12488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:14.351584 12488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:14.364609 12493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:14.364640 12496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:14.364677 12494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:14.365607 12488 server_base.cc:1034] running on GCE node
I20260929 02:17:14.366127 12488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:14.367084 12488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:14.368270 12488 hybrid_clock.cc:648] HybridClock initialized: now 1790648234368218 us; error 49 us; skew 500 ppm
I20260929 02:17:14.368525 12488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:14 dist-test-slave-9k0h krb5kdc[12248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648234, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:14.371749 12488 init.cc:377] Logged in from keytab as kudu/127.24.216.130@KRBTEST.COM (short username kudu)
I20260929 02:17:14.373160 12488 webserver.cc:460] Webserver started at http://127.24.216.130:44267/ using document root <none> and password file <none>
I20260929 02:17:14.373822 12488 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:14.373970 12488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:14.374351 12488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:14.377115 12488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "03c3857970994f4a844f3ec53617e401"
format_stamp: "Formatted at 2026-09-29 02:17:14 on dist-test-slave-9k0h"
server_key: "d6edede81f292a8b0336ba4aa23cb548"
server_key_iv: "2bf961b9846cd0dda2f7d1b166048873"
server_key_version: "encryptionkey@0"
I20260929 02:17:14.377866 12488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "03c3857970994f4a844f3ec53617e401"
format_stamp: "Formatted at 2026-09-29 02:17:14 on dist-test-slave-9k0h"
server_key: "d6edede81f292a8b0336ba4aa23cb548"
server_key_iv: "2bf961b9846cd0dda2f7d1b166048873"
server_key_version: "encryptionkey@0"
I20260929 02:17:14.382961 12488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:17:14.386324 12503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:14.387537 12488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:17:14.387708 12488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "03c3857970994f4a844f3ec53617e401"
format_stamp: "Formatted at 2026-09-29 02:17:14 on dist-test-slave-9k0h"
server_key: "d6edede81f292a8b0336ba4aa23cb548"
server_key_iv: "2bf961b9846cd0dda2f7d1b166048873"
server_key_version: "encryptionkey@0"
I20260929 02:17:14.387894 12488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:14.403435 12488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:14.407341 12488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:14.407618 12488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:14.408510 12488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:14.409763 12488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:14.409871 12488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:14.410020 12488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:14.410092 12488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:14.441936 12488 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:40405
I20260929 02:17:14.441957 12615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:40405 every 8 connection(s)
I20260929 02:17:14.443261 12488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:17:14.451421 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 12488
I20260929 02:17:14.451609 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:17:14.452032 25442 external_mini_cluster.cc:1428] Setting key fcc7c7c2350300a1291c906088169f62
Sep 29 02:17:14 dist-test-slave-9k0h krb5kdc[12248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648234, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.130@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:17:14.463614 12484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:14.445761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:39973 (local address 127.24.216.190:37475)
0929 02:17:14.445967 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:14.445974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:14.447056 (+ 1082us) server_negotiation.cc:407] Connection header received
0929 02:17:14.448740 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:14.448744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:14.448814 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:14.448900 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:14.450289 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:14.450978 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:14.452596 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:14.453083 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:14.453286 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:14.456826 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:14.456831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:14.456834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:14.456865 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:14.459588 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:14.460512 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:14.460517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:14.460520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:14.460686 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:14.461236 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:14.461243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:14.461249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:14.461653 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:14.461814 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:14.462856 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0929 02:17:14.463171 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":68}
I20260929 02:17:14.464315 12618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:14.446240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:37475 (local address 127.24.216.130:39973)
0929 02:17:14.446904 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:14.446946 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:14.448507 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:14.449098 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:14.449105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:14.449593 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:14.450127 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:14.450160 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:14.451104 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:14.451108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:14.452452 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:14.452463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:14.453364 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:14.453374 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:14.453607 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:14.454301 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:14.454323 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:14.456612 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:14.459757 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:14.459766 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:14.459781 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:14.460374 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:14.460833 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:14.460839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:14.460843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:14.461089 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:14.461824 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:14.461832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:14.462358 (+ 526us) client_negotiation.cc:769] Sending connection context
0929 02:17:14.462659 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 02:17:14.463192 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":253,"threads_started":1}
I20260929 02:17:14.465785 12616 heartbeater.cc:347] Connected to a master server at 127.24.216.190:37475
I20260929 02:17:14.466074 12616 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:14.466861 12616 heartbeater.cc:511] Master 127.24.216.190:37475 requested a full tablet report, sending...
I20260929 02:17:14.468374 12293 ts_manager.cc:194] Registered new tserver with Master: 03c3857970994f4a844f3ec53617e401 (127.24.216.130:40405)
I20260929 02:17:14.469172 12293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.130@KRBTEST.COM'} at 127.24.216.130:39973
WARNING: no policy specified for kudu/127.24.216.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
Entry for principal kudu/127.24.216.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab.
I20260929 02:17:14.509042 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37475
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:14.837462 12621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:14.837809 12621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:14.837991 12621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:14.845582 12621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:14.845747 12621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:14.845944 12621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:17:14.856302 12621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.131.keytab
--principal=kudu/127.24.216.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:14.857895 12621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:14.859355 12621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:14.873015 12626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:14.873085 12627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:14.873108 12629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:14.873859 12621 server_base.cc:1034] running on GCE node
I20260929 02:17:14.874562 12621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:14.875507 12621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:14.876747 12621 hybrid_clock.cc:648] HybridClock initialized: now 1790648234876690 us; error 46 us; skew 500 ppm
I20260929 02:17:14.877187 12621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:14 dist-test-slave-9k0h krb5kdc[12248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648234, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:14.888065 12621 init.cc:377] Logged in from keytab as kudu/127.24.216.131@KRBTEST.COM (short username kudu)
I20260929 02:17:14.889715 12621 webserver.cc:460] Webserver started at http://127.24.216.131:37489/ using document root <none> and password file <none>
I20260929 02:17:14.890539 12621 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:14.890690 12621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:14.891069 12621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:14.894098 12621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "a0b80fb84c8547399d769a5ed2e105e3"
format_stamp: "Formatted at 2026-09-29 02:17:14 on dist-test-slave-9k0h"
server_key: "4f79338d53dddaaa8bb55834c3eb1f17"
server_key_iv: "478efaf35552fc18e9a4941280b47a60"
server_key_version: "encryptionkey@0"
I20260929 02:17:14.894935 12621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "a0b80fb84c8547399d769a5ed2e105e3"
format_stamp: "Formatted at 2026-09-29 02:17:14 on dist-test-slave-9k0h"
server_key: "4f79338d53dddaaa8bb55834c3eb1f17"
server_key_iv: "478efaf35552fc18e9a4941280b47a60"
server_key_version: "encryptionkey@0"
I20260929 02:17:14.900179 12621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:17:14.903745 12636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:14.905123 12621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:17:14.905323 12621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "a0b80fb84c8547399d769a5ed2e105e3"
format_stamp: "Formatted at 2026-09-29 02:17:14 on dist-test-slave-9k0h"
server_key: "4f79338d53dddaaa8bb55834c3eb1f17"
server_key_iv: "478efaf35552fc18e9a4941280b47a60"
server_key_version: "encryptionkey@0"
I20260929 02:17:14.905529 12621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:14.921623 12621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:14.925976 12621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:14.926332 12621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:14.927367 12621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:14.928714 12621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:14.928817 12621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:14.928951 12621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:14.929031 12621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:14.957499 12482 heartbeater.cc:503] Master 127.24.216.190:37475 was elected leader, sending a full tablet report...
I20260929 02:17:14.962033 12621 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:35765
I20260929 02:17:14.962074 12748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:35765 every 8 connection(s)
I20260929 02:17:14.963315 12621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:17:14.964330 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 12621
I20260929 02:17:14.964524 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:17:14.965106 25442 external_mini_cluster.cc:1428] Setting key 655319a779f7f080a19f721ee9c1353d
Sep 29 02:17:14 dist-test-slave-9k0h krb5kdc[12248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648234, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.131@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:17:14.983618 12751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:14.966415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:50629 (local address 127.24.216.190:37475)
0929 02:17:14.966901 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:14.966907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:14.968163 (+ 1256us) server_negotiation.cc:407] Connection header received
0929 02:17:14.969957 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:14.969961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:14.970025 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:14.970127 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:14.971673 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:14.972528 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:14.974120 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:14.974615 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:14.974732 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:14.978062 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:14.978076 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:14.978080 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:14.978113 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:14.980992 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:14.981641 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:14.981645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:14.981647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:14.981762 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:14.982125 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:14.982128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:14.982130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:14.982431 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:14.982541 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:14.983160 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 02:17:14.983314 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":268,"threads_started":1}
I20260929 02:17:14.984552 12752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:14.967096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:37475 (local address 127.24.216.131:50629)
0929 02:17:14.968000 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:14.968058 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:14.969710 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:14.970378 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:14.970386 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:14.970897 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:14.971471 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:14.971489 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:14.972685 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:14.972689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:14.973968 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:14.973979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:14.974826 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:14.974830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:14.975026 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:14.975626 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:14.975647 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:14.977866 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:14.981121 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:14.981127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:14.981139 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:14.981537 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:14.981854 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:14.981857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:14.981859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:14.982035 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:14.982536 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:14.982542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:14.982852 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 02:17:14.983082 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 02:17:14.983451 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":327,"threads_started":1}
I20260929 02:17:14.985905 12749 heartbeater.cc:347] Connected to a master server at 127.24.216.190:37475
I20260929 02:17:14.986233 12749 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:14.986934 12749 heartbeater.cc:511] Master 127.24.216.190:37475 requested a full tablet report, sending...
I20260929 02:17:14.988389 12293 ts_manager.cc:194] Registered new tserver with Master: a0b80fb84c8547399d769a5ed2e105e3 (127.24.216.131:35765)
I20260929 02:17:14.989265 12293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.216.131@KRBTEST.COM'} at 127.24.216.131:50629
I20260929 02:17:14.992877 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:17:15.009856 12751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:14.996914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.24.216.190:37475)
0929 02:17:14.997113 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:14.997120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:14.997625 (+ 505us) server_negotiation.cc:407] Connection header received
0929 02:17:14.997759 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:14.997763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:14.997827 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:14.997908 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:14.999331 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.000103 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.001356 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.001758 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.001855 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.004183 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:15.004189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:15.004192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:15.004227 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:15.007210 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:15.007798 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:15.007802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:15.007804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:15.007935 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:15.008252 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:15.008256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:15.008258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:15.008688 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:15.008949 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:15.009219 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 02:17:15.009496 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:17:15.014026 12293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:17:15.020937 12293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:17:15.066399 12764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:15.032316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:40405 (local address 127.0.0.1:58552)
0929 02:17:15.044737 (+ 12421us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:15.044780 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:15.044992 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:15.058786 (+ 13794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:15.058793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:15.058814 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:15.059315 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:15.059334 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.060670 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.060678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:15.063899 (+ 3221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:15.063914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.065252 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.065258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:15.065445 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.065503 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 02:17:15.065660 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 02:17:15.065853 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12138,"mutex_wait_us":430,"thread_start_us":24662,"threads_started":1}
I20260929 02:17:15.066547 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:15.033931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.24.216.129:41777)
0929 02:17:15.056274 (+ 22343us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:15.056287 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:15.057733 (+ 1446us) server_negotiation.cc:407] Connection header received
0929 02:17:15.057857 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:15.057869 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:15.058203 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:15.058431 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:15.059584 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.061081 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.063815 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.065286 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.065499 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.065660 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:15.065983 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 02:17:15.066203 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22110,"thread_start_us":22624,"threads_started":1}
I20260929 02:17:15.067308 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:15.033172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.24.216.131:35765)
0929 02:17:15.055794 (+ 22622us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:15.055809 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:15.057940 (+ 2131us) server_negotiation.cc:407] Connection header received
0929 02:17:15.058038 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:15.058044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:15.058249 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:15.058431 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:15.059112 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.060835 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.064055 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.065390 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.065550 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.065697 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:15.066800 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0929 02:17:15.066949 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22351,"thread_start_us":22633,"threads_started":1}
I20260929 02:17:15.067356 12762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:15.033185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:41777 (local address 127.0.0.1:55438)
0929 02:17:15.057373 (+ 24188us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:15.057400 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:15.057587 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:15.059016 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:15.059024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:15.059056 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:15.059415 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:15.059423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.061213 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.061220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:15.063651 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:15.063668 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.065621 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.065630 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:15.065800 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.065833 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:17:15.065924 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 02:17:15.066076 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24001,"mutex_wait_us":4,"thread_start_us":23129,"threads_started":1}
I20260929 02:17:15.067684 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:15.032533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:35765 (local address 127.0.0.1:58698)
0929 02:17:15.057671 (+ 25138us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:15.057690 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:15.057815 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:15.058451 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:15.058458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:15.058488 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:15.058905 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:15.058919 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.060970 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.060975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:15.063899 (+ 2924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:15.063914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.065830 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.065834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:15.066052 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.066087 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:17:15.066269 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 02:17:15.066426 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23951,"mutex_wait_us":517,"thread_start_us":11772,"threads_started":1}
I20260929 02:17:15.067745 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:15.032524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.24.216.130:40405)
0929 02:17:15.058123 (+ 25599us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:15.058129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:15.058203 (+ 74us) server_negotiation.cc:407] Connection header received
0929 02:17:15.058285 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:15.058290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:15.058512 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:15.058777 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:15.059539 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.060534 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.064101 (+ 3567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.065039 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.065230 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.065464 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:15.067256 (+ 1792us) server_negotiation.cc:299] Negotiation successful
0929 02:17:15.067440 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25447,"thread_start_us":25352,"threads_started":1}
I20260929 02:17:15.071198 12417 tablet_service.cc:1467] Processing CreateTablet for tablet 2504745504bb46518e359bf45604cb58 (DEFAULT_TABLE table=test-table [id=27d9135a01004d56b370f9ac45964bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:15.071239 12551 tablet_service.cc:1467] Processing CreateTablet for tablet 2504745504bb46518e359bf45604cb58 (DEFAULT_TABLE table=test-table [id=27d9135a01004d56b370f9ac45964bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:15.071751 12684 tablet_service.cc:1467] Processing CreateTablet for tablet 2504745504bb46518e359bf45604cb58 (DEFAULT_TABLE table=test-table [id=27d9135a01004d56b370f9ac45964bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:15.072592 12417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2504745504bb46518e359bf45604cb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:15.072629 12551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2504745504bb46518e359bf45604cb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:15.073000 12684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2504745504bb46518e359bf45604cb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:15.085125 12768 tablet_bootstrap.cc:492] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b: Bootstrap starting.
I20260929 02:17:15.087011 12769 tablet_bootstrap.cc:492] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3: Bootstrap starting.
I20260929 02:17:15.088651 12768 tablet_bootstrap.cc:654] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:15.088671 12770 tablet_bootstrap.cc:492] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401: Bootstrap starting.
I20260929 02:17:15.089982 12768 log.cc:824] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:15.090881 12769 tablet_bootstrap.cc:654] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:15.091917 12770 tablet_bootstrap.cc:654] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:15.092216 12769 log.cc:824] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:15.093119 12770 log.cc:824] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:15.093523 12768 tablet_bootstrap.cc:492] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b: No bootstrap required, opened a new log
I20260929 02:17:15.093840 12768 ts_tablet_manager.cc:1397] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260929 02:17:15.095360 12769 tablet_bootstrap.cc:492] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3: No bootstrap required, opened a new log
I20260929 02:17:15.095619 12769 ts_tablet_manager.cc:1397] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:17:15.096315 12770 tablet_bootstrap.cc:492] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401: No bootstrap required, opened a new log
I20260929 02:17:15.096562 12770 ts_tablet_manager.cc:1397] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 02:17:15.102573 12769 raft_consensus.cc:348] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c3857970994f4a844f3ec53617e401" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40405 } } peers { permanent_uuid: "a0b80fb84c8547399d769a5ed2e105e3" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35765 } } peers { permanent_uuid: "5990271b7a6a4d75acf823ebedefd33b" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41777 } }
I20260929 02:17:15.102938 12769 raft_consensus.cc:374] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:15.103031 12769 raft_consensus.cc:729] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b80fb84c8547399d769a5ed2e105e3, State: Initialized, Role: FOLLOWER
I20260929 02:17:15.103642 12769 consensus_queue.cc:260] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c3857970994f4a844f3ec53617e401" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40405 } } peers { permanent_uuid: "a0b80fb84c8547399d769a5ed2e105e3" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35765 } } peers { permanent_uuid: "5990271b7a6a4d75acf823ebedefd33b" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41777 } }
I20260929 02:17:15.104738 12749 heartbeater.cc:503] Master 127.24.216.190:37475 was elected leader, sending a full tablet report...
I20260929 02:17:15.104703 12768 raft_consensus.cc:348] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c3857970994f4a844f3ec53617e401" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40405 } } peers { permanent_uuid: "a0b80fb84c8547399d769a5ed2e105e3" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35765 } } peers { permanent_uuid: "5990271b7a6a4d75acf823ebedefd33b" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41777 } }
I20260929 02:17:15.105108 12768 raft_consensus.cc:374] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:15.105204 12768 raft_consensus.cc:729] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5990271b7a6a4d75acf823ebedefd33b, State: Initialized, Role: FOLLOWER
I20260929 02:17:15.105365 12769 ts_tablet_manager.cc:1428] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260929 02:17:15.105863 12768 consensus_queue.cc:260] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c3857970994f4a844f3ec53617e401" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40405 } } peers { permanent_uuid: "a0b80fb84c8547399d769a5ed2e105e3" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35765 } } peers { permanent_uuid: "5990271b7a6a4d75acf823ebedefd33b" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41777 } }
I20260929 02:17:15.107053 12770 raft_consensus.cc:348] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c3857970994f4a844f3ec53617e401" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40405 } } peers { permanent_uuid: "a0b80fb84c8547399d769a5ed2e105e3" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35765 } } peers { permanent_uuid: "5990271b7a6a4d75acf823ebedefd33b" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41777 } }
I20260929 02:17:15.107486 12770 raft_consensus.cc:374] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:15.107579 12770 raft_consensus.cc:729] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c3857970994f4a844f3ec53617e401, State: Initialized, Role: FOLLOWER
I20260929 02:17:15.107684 12768 ts_tablet_manager.cc:1428] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260929 02:17:15.108150 12770 consensus_queue.cc:260] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c3857970994f4a844f3ec53617e401" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40405 } } peers { permanent_uuid: "a0b80fb84c8547399d769a5ed2e105e3" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35765 } } peers { permanent_uuid: "5990271b7a6a4d75acf823ebedefd33b" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41777 } }
I20260929 02:17:15.111405 12770 ts_tablet_manager.cc:1428] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260929 02:17:15.111557 12616 heartbeater.cc:503] Master 127.24.216.190:37475 was elected leader, sending a full tablet report...
W20260929 02:17:15.178260 12483 tablet.cc:2369] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:17:15.192509 12776 raft_consensus.cc:482] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:17:15.192826 12776 raft_consensus.cc:504] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c3857970994f4a844f3ec53617e401" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40405 } } peers { permanent_uuid: "a0b80fb84c8547399d769a5ed2e105e3" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35765 } } peers { permanent_uuid: "5990271b7a6a4d75acf823ebedefd33b" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41777 } }
I20260929 02:17:15.194561 12776 leader_election.cc:288] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0b80fb84c8547399d769a5ed2e105e3 (127.24.216.131:35765), 5990271b7a6a4d75acf823ebedefd33b (127.24.216.129:41777)
W20260929 02:17:15.195264 12617 tablet.cc:2369] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:17:15.201963 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:15.195001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:47131 (local address 127.24.216.131:35765)
0929 02:17:15.195580 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:15.195589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:15.195755 (+ 166us) server_negotiation.cc:407] Connection header received
0929 02:17:15.196386 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:15.196390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:15.196452 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:15.196550 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:15.197109 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.197860 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.200232 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.200974 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.201162 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.201244 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:15.201576 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 02:17:15.201697 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428}
I20260929 02:17:15.201987 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:15.195750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:34939 (local address 127.24.216.129:41777)
0929 02:17:15.195987 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:15.195993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:15.196383 (+ 390us) server_negotiation.cc:407] Connection header received
0929 02:17:15.196576 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:15.196581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:15.196637 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:15.196735 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:15.197326 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.198006 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.200169 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.200970 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.201153 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.201233 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:15.201576 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 02:17:15.201708 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:17:15.202106 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:15.194751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:35765 (local address 127.24.216.130:47131)
0929 02:17:15.195603 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:15.195650 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:15.196118 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:15.196568 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:15.196573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:15.196600 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:15.196963 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:15.196975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.198001 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.198007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:15.200047 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:15.200060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.201241 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.201247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:15.201412 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.201440 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:17:15.201540 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:17:15.201688 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":46,"thread_start_us":518,"threads_started":1}
I20260929 02:17:15.202474 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:15.195079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:41777 (local address 127.24.216.130:34939)
0929 02:17:15.196283 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:15.196297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:15.196443 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:15.196765 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:15.196773 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:15.196799 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:15.197161 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:15.197169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.198144 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.198169 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:15.200016 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:15.200034 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.201211 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.201218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:15.201379 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.201447 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 02:17:15.201540 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 02:17:15.201669 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":114,"thread_start_us":833,"threads_started":1}
I20260929 02:17:15.203157 12704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2504745504bb46518e359bf45604cb58" candidate_uuid: "03c3857970994f4a844f3ec53617e401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b80fb84c8547399d769a5ed2e105e3" is_pre_election: true
I20260929 02:17:15.203382 12437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2504745504bb46518e359bf45604cb58" candidate_uuid: "03c3857970994f4a844f3ec53617e401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5990271b7a6a4d75acf823ebedefd33b" is_pre_election: true
I20260929 02:17:15.203780 12704 raft_consensus.cc:2454] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03c3857970994f4a844f3ec53617e401 in term 0.
I20260929 02:17:15.203895 12437 raft_consensus.cc:2454] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03c3857970994f4a844f3ec53617e401 in term 0.
I20260929 02:17:15.204479 12504 leader_election.cc:302] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03c3857970994f4a844f3ec53617e401, a0b80fb84c8547399d769a5ed2e105e3; no voters:
I20260929 02:17:15.204882 12776 raft_consensus.cc:2781] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:17:15.205026 12776 raft_consensus.cc:482] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:17:15.205155 12776 raft_consensus.cc:3037] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:15.207444 12776 raft_consensus.cc:504] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c3857970994f4a844f3ec53617e401" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40405 } } peers { permanent_uuid: "a0b80fb84c8547399d769a5ed2e105e3" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35765 } } peers { permanent_uuid: "5990271b7a6a4d75acf823ebedefd33b" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41777 } }
I20260929 02:17:15.208046 12776 leader_election.cc:288] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [CANDIDATE]: Term 1 election: Requested vote from peers a0b80fb84c8547399d769a5ed2e105e3 (127.24.216.131:35765), 5990271b7a6a4d75acf823ebedefd33b (127.24.216.129:41777)
I20260929 02:17:15.208489 12704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2504745504bb46518e359bf45604cb58" candidate_uuid: "03c3857970994f4a844f3ec53617e401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b80fb84c8547399d769a5ed2e105e3"
I20260929 02:17:15.208539 12437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2504745504bb46518e359bf45604cb58" candidate_uuid: "03c3857970994f4a844f3ec53617e401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5990271b7a6a4d75acf823ebedefd33b"
I20260929 02:17:15.208722 12437 raft_consensus.cc:3037] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:15.208722 12704 raft_consensus.cc:3037] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:15.211962 12437 raft_consensus.cc:2454] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03c3857970994f4a844f3ec53617e401 in term 1.
I20260929 02:17:15.211962 12704 raft_consensus.cc:2454] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03c3857970994f4a844f3ec53617e401 in term 1.
I20260929 02:17:15.212356 12505 leader_election.cc:302] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03c3857970994f4a844f3ec53617e401, 5990271b7a6a4d75acf823ebedefd33b; no voters:
I20260929 02:17:15.212647 12776 raft_consensus.cc:2781] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:15.213071 12776 raft_consensus.cc:686] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [term 1 LEADER]: Becoming Leader. State: Replica: 03c3857970994f4a844f3ec53617e401, State: Running, Role: LEADER
I20260929 02:17:15.213590 12776 consensus_queue.cc:237] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c3857970994f4a844f3ec53617e401" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40405 } } peers { permanent_uuid: "a0b80fb84c8547399d769a5ed2e105e3" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35765 } } peers { permanent_uuid: "5990271b7a6a4d75acf823ebedefd33b" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41777 } }
W20260929 02:17:15.215252 12750 tablet.cc:2369] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:17:15.222975 12293 catalog_manager.cc:5495] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03c3857970994f4a844f3ec53617e401 (127.24.216.130). New cstate: current_term: 1 leader_uuid: "03c3857970994f4a844f3ec53617e401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c3857970994f4a844f3ec53617e401" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 40405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0b80fb84c8547399d769a5ed2e105e3" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5990271b7a6a4d75acf823ebedefd33b" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41777 } health_report { overall_health: UNKNOWN } } }
I20260929 02:17:15.248170 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:15.243120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58558 (local address 127.24.216.130:40405)
0929 02:17:15.243332 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:15.243338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:15.243434 (+ 96us) server_negotiation.cc:407] Connection header received
0929 02:17:15.243730 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:15.243734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:15.243815 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:15.243935 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:15.244242 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.244932 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.246254 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.246639 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.246730 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.247125 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:15.247610 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:15.247714 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:15.247800 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 02:17:15.247862 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:17:15.256206 12437 raft_consensus.cc:1261] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b [term 1 FOLLOWER]: Refusing update from remote peer 03c3857970994f4a844f3ec53617e401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:17:15.256218 12704 raft_consensus.cc:1261] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3 [term 1 FOLLOWER]: Refusing update from remote peer 03c3857970994f4a844f3ec53617e401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:17:15.257207 12779 consensus_queue.cc:1035] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5990271b7a6a4d75acf823ebedefd33b" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 41777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:17:15.257530 12776 consensus_queue.cc:1035] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b80fb84c8547399d769a5ed2e105e3" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 35765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:17:15.284799 12292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:17:15.285104 12292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:17:15.289089 12292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 448a0a64f2ac4e848118c8f81d5a86a0: Sending DeleteTablet for 3 replicas of tablet 2504745504bb46518e359bf45604cb58
I20260929 02:17:15.290360 12417 tablet_service.cc:1514] Processing DeleteTablet for tablet 2504745504bb46518e359bf45604cb58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:15 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:55438
I20260929 02:17:15.290341 12551 tablet_service.cc:1514] Processing DeleteTablet for tablet 2504745504bb46518e359bf45604cb58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:15 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:58552
I20260929 02:17:15.290390 12684 tablet_service.cc:1514] Processing DeleteTablet for tablet 2504745504bb46518e359bf45604cb58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:15 UTC) from {username='kudu', principal='kudu/127.24.216.190@KRBTEST.COM'} at 127.0.0.1:58698
I20260929 02:17:15.293144 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 12354
I20260929 02:17:15.301436 12792 tablet_replica.cc:330] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b: stopping tablet replica
I20260929 02:17:15.301934 12794 tablet_replica.cc:330] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3: stopping tablet replica
I20260929 02:17:15.301995 12792 raft_consensus.cc:2229] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:17:15.302503 12794 raft_consensus.cc:2229] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:17:15.302637 12792 raft_consensus.cc:2258] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:15.303097 12794 raft_consensus.cc:2258] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:15.305348 12794 ts_tablet_manager.cc:1905] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:15.305351 12792 ts_tablet_manager.cc:1905] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:15.311065 12793 tablet_replica.cc:330] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401: stopping tablet replica
I20260929 02:17:15.311532 12793 raft_consensus.cc:2229] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:17:15.312212 12793 raft_consensus.cc:2258] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:15.312947 12792 ts_tablet_manager.cc:1918] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:15.313109 12792 log.cc:1196] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/2504745504bb46518e359bf45604cb58
I20260929 02:17:15.313141 12794 ts_tablet_manager.cc:1918] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:15.313429 12794 log.cc:1196] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/2504745504bb46518e359bf45604cb58
I20260929 02:17:15.313649 12792 ts_tablet_manager.cc:1939] T 2504745504bb46518e359bf45604cb58 P 5990271b7a6a4d75acf823ebedefd33b: Deleting consensus metadata
I20260929 02:17:15.314070 12794 ts_tablet_manager.cc:1939] T 2504745504bb46518e359bf45604cb58 P a0b80fb84c8547399d769a5ed2e105e3: Deleting consensus metadata
I20260929 02:17:15.315661 12278 catalog_manager.cc:4841] TS 5990271b7a6a4d75acf823ebedefd33b (127.24.216.129:41777): tablet 2504745504bb46518e359bf45604cb58 (table test-table [id=27d9135a01004d56b370f9ac45964bbe]) successfully deleted
Sep 29 02:17:15 dist-test-slave-9k0h krb5kdc[12248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.129@KRBTEST.COM
I20260929 02:17:15.315830 12277 catalog_manager.cc:4841] TS a0b80fb84c8547399d769a5ed2e105e3 (127.24.216.131:35765): tablet 2504745504bb46518e359bf45604cb58 (table test-table [id=27d9135a01004d56b370f9ac45964bbe]) successfully deleted
I20260929 02:17:15.317510 12793 ts_tablet_manager.cc:1905] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:15.323268 12793 ts_tablet_manager.cc:1918] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:15.323446 12793 log.cc:1196] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/2504745504bb46518e359bf45604cb58
I20260929 02:17:15.324028 12793 ts_tablet_manager.cc:1939] T 2504745504bb46518e359bf45604cb58 P 03c3857970994f4a844f3ec53617e401: Deleting consensus metadata
I20260929 02:17:15.324451 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:15.294951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.24.216.129:41777)
0929 02:17:15.295181 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:15.295190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:15.303787 (+ 8597us) server_negotiation.cc:407] Connection header received
0929 02:17:15.304079 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:15.304085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:15.304168 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:15.304258 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:15.306064 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.307250 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.309146 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.309814 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.309935 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.317183 (+ 7248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:15.317197 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:15.317214 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:15.317263 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:15.321100 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:15.321952 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:15.321963 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:15.321967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:15.322129 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:15.322590 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:15.322595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:15.322599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:15.323051 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:15.323340 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:15.323771 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 02:17:15.323990 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:17:15.325139 12477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:15.325810 12280 catalog_manager.cc:4841] TS 03c3857970994f4a844f3ec53617e401 (127.24.216.130:40405): tablet 2504745504bb46518e359bf45604cb58 (table test-table [id=27d9135a01004d56b370f9ac45964bbe]) successfully deleted
I20260929 02:17:15.424413 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 12354
I20260929 02:17:15.442247 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 12488
Sep 29 02:17:15 dist-test-slave-9k0h krb5kdc[12248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.130@KRBTEST.COM
I20260929 02:17:15.458413 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:15.443411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58562 (local address 127.24.216.130:40405)
0929 02:17:15.443581 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:15.443588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:15.443635 (+ 47us) server_negotiation.cc:407] Connection header received
0929 02:17:15.443743 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:15.443752 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:15.443876 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:15.444006 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:15.445220 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.446048 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.447548 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.448023 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.448168 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.452369 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:15.452380 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:15.452394 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:15.452430 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:15.455614 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:15.456295 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:15.456302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:15.456304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:15.456448 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:15.456809 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:15.456815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:15.456820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:15.457196 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:15.457540 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:15.457773 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 02:17:15.457989 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:17:15.459233 12611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:15.576076 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 12488
I20260929 02:17:15.597470 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 12621
Sep 29 02:17:15 dist-test-slave-9k0h krb5kdc[12248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.131@KRBTEST.COM
I20260929 02:17:15.612708 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:15.598348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58712 (local address 127.24.216.131:35765)
0929 02:17:15.598531 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:15.598535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:15.598554 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:17:15.598656 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:15.598660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:15.598730 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:15.598812 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:15.600064 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.600776 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.602011 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:15.602455 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:15.602547 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:15.607051 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:15.607074 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:15.607097 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:15.607149 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:15.609986 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:15.610730 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:15.610737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:15.610739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:15.610878 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:15.611295 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:15.611298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:15.611300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:15.611642 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:15.611859 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:15.612159 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 02:17:15.612329 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:17:15.613444 12744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:15.704284 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 12621
I20260929 02:17:15.721866 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 12261
I20260929 02:17:15.722956 12323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:15.831199 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 12261
2026-09-29T02:17:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4318 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T02:17:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:15Z Disabled control of system clock
I20260929 02:17:15.912446 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:35717
--webserver_interface=127.24.216.190
--webserver_port=0
--builtin_ntp_servers=127.24.216.148:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:35717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:17:16.235227 12803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:16.235496 12803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:16.235576 12803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:16.242448 12803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:16.242547 12803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:16.242584 12803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:16.242640 12803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:16.242683 12803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:16.251543 12803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:35717
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:35717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:16.252805 12803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:16.254462 12803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:16.265785 12808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:16.265820 12811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:16.265861 12809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:16.266990 12803 server_base.cc:1034] running on GCE node
I20260929 02:17:16.267656 12803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:16.268870 12803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:16.270092 12803 hybrid_clock.cc:648] HybridClock initialized: now 1790648236270021 us; error 50 us; skew 500 ppm
I20260929 02:17:16.270443 12803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:17:16.272660 12803 webserver.cc:460] Webserver started at http://127.24.216.190:46571/ using document root <none> and password file <none>
I20260929 02:17:16.273260 12803 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:16.273342 12803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:16.273592 12803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:16.276371 12803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "c76e51c85a8c4d1cb84f35bccd6db766"
format_stamp: "Formatted at 2026-09-29 02:17:16 on dist-test-slave-9k0h"
server_key: "bbc11863da156139dc60253f2278a2f8"
server_key_iv: "1937f85c54f87533a68203df3925ea81"
server_key_version: "encryptionkey@0"
I20260929 02:17:16.277089 12803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "c76e51c85a8c4d1cb84f35bccd6db766"
format_stamp: "Formatted at 2026-09-29 02:17:16 on dist-test-slave-9k0h"
server_key: "bbc11863da156139dc60253f2278a2f8"
server_key_iv: "1937f85c54f87533a68203df3925ea81"
server_key_version: "encryptionkey@0"
I20260929 02:17:16.281960 12803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260929 02:17:16.285198 12817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:16.286510 12803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:17:16.286690 12803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "c76e51c85a8c4d1cb84f35bccd6db766"
format_stamp: "Formatted at 2026-09-29 02:17:16 on dist-test-slave-9k0h"
server_key: "bbc11863da156139dc60253f2278a2f8"
server_key_iv: "1937f85c54f87533a68203df3925ea81"
server_key_version: "encryptionkey@0"
I20260929 02:17:16.286888 12803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:16.301824 12803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:16.302673 12803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:16.302958 12803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:16.320955 12803 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:35717
I20260929 02:17:16.320974 12868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:35717 every 8 connection(s)
I20260929 02:17:16.322273 12803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:16.325778 12869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:16.326632 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 12803
I20260929 02:17:16.326797 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:16.327255 25442 external_mini_cluster.cc:1428] Setting key 91eb3249f03f4b13f64a0f15085288d2
I20260929 02:17:16.339746 12869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766: Bootstrap starting.
I20260929 02:17:16.342105 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:16.329142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.24.216.190:35717)
0929 02:17:16.330141 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:16.330185 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:16.330246 (+ 61us) server_negotiation.cc:407] Connection header received
0929 02:17:16.332402 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:16.332429 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:16.332998 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:16.333446 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:16.333946 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:16.335780 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:16.337134 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:16.338204 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:16.338364 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:16.339394 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:16.339404 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:16.339422 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:16.340104 (+ 682us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:16.340207 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:16.340231 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:16.340345 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:16.340567 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 02:17:16.340854 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":339,"threads_started":1}
I20260929 02:17:16.343377 12869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:16.344332 12869 log.cc:824] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:16.347157 12869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766: No bootstrap required, opened a new log
I20260929 02:17:16.354393 12869 raft_consensus.cc:348] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76e51c85a8c4d1cb84f35bccd6db766" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35717 } }
I20260929 02:17:16.354765 12869 raft_consensus.cc:374] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:16.354907 12869 raft_consensus.cc:729] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76e51c85a8c4d1cb84f35bccd6db766, State: Initialized, Role: FOLLOWER
I20260929 02:17:16.355492 12869 consensus_queue.cc:260] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76e51c85a8c4d1cb84f35bccd6db766" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35717 } }
I20260929 02:17:16.355786 12869 raft_consensus.cc:388] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:16.355950 12869 raft_consensus.cc:482] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:16.356133 12869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:16.358292 12869 raft_consensus.cc:504] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76e51c85a8c4d1cb84f35bccd6db766" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35717 } }
I20260929 02:17:16.358757 12869 leader_election.cc:302] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c76e51c85a8c4d1cb84f35bccd6db766; no voters:
I20260929 02:17:16.359288 12869 leader_election.cc:288] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:16.359423 12874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:16.360108 12874 raft_consensus.cc:686] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [term 1 LEADER]: Becoming Leader. State: Replica: c76e51c85a8c4d1cb84f35bccd6db766, State: Running, Role: LEADER
I20260929 02:17:16.360556 12874 consensus_queue.cc:237] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76e51c85a8c4d1cb84f35bccd6db766" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35717 } }
I20260929 02:17:16.361227 12869 sys_catalog.cc:564] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:16.363128 12875 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c76e51c85a8c4d1cb84f35bccd6db766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76e51c85a8c4d1cb84f35bccd6db766" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35717 } } }
I20260929 02:17:16.363152 12876 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c76e51c85a8c4d1cb84f35bccd6db766. Latest consensus state: current_term: 1 leader_uuid: "c76e51c85a8c4d1cb84f35bccd6db766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76e51c85a8c4d1cb84f35bccd6db766" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35717 } } }
I20260929 02:17:16.363734 12876 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:16.363600 12875 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:16.364771 12879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:16.371030 12879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:16.381249 12879 catalog_manager.cc:1347] Generated new cluster ID: e7777ce628f0485a962cf915857f38d8
I20260929 02:17:16.381354 12879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:16.403265 12879 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:16.405019 12879 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:16.420095 12879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766: Generated new TSK 0
I20260929 02:17:16.421109 12879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:16.425804 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.129:0
--local_ip_for_outbound_sockets=127.24.216.129
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35717
--builtin_ntp_servers=127.24.216.148:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:17:16.755743 12893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:16.756011 12893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:16.756096 12893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:16.763485 12893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:16.763581 12893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:16.763772 12893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.129
I20260929 02:17:16.772756 12893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.216.129
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.129
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:16.774046 12893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:16.775805 12893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:16.788642 12898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:16.788586 12899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:16.788734 12901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:16.789606 12893 server_base.cc:1034] running on GCE node
I20260929 02:17:16.790257 12893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:16.791215 12893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:16.792431 12893 hybrid_clock.cc:648] HybridClock initialized: now 1790648236792388 us; error 36 us; skew 500 ppm
I20260929 02:17:16.792738 12893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:17:16.794915 12893 webserver.cc:460] Webserver started at http://127.24.216.129:41667/ using document root <none> and password file <none>
I20260929 02:17:16.795538 12893 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:16.795650 12893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:16.795965 12893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:16.798799 12893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/data/instance:
uuid: "da51a9f512f548c0a5a425341bf27d94"
format_stamp: "Formatted at 2026-09-29 02:17:16 on dist-test-slave-9k0h"
server_key: "025a503188fefb015a601a63a3571872"
server_key_iv: "943ec177bdbfc9aeaf6b670f9e632293"
server_key_version: "encryptionkey@0"
I20260929 02:17:16.799552 12893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance:
uuid: "da51a9f512f548c0a5a425341bf27d94"
format_stamp: "Formatted at 2026-09-29 02:17:16 on dist-test-slave-9k0h"
server_key: "025a503188fefb015a601a63a3571872"
server_key_iv: "943ec177bdbfc9aeaf6b670f9e632293"
server_key_version: "encryptionkey@0"
I20260929 02:17:16.804221 12893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:17:16.807492 12907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:16.808679 12893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 02:17:16.808840 12893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/wal
uuid: "da51a9f512f548c0a5a425341bf27d94"
format_stamp: "Formatted at 2026-09-29 02:17:16 on dist-test-slave-9k0h"
server_key: "025a503188fefb015a601a63a3571872"
server_key_iv: "943ec177bdbfc9aeaf6b670f9e632293"
server_key_version: "encryptionkey@0"
I20260929 02:17:16.809059 12893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:16.823046 12893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:16.823768 12893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:16.824049 12893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:16.824913 12893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:16.826318 12893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:16.826418 12893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:16.826534 12893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:16.826630 12893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:16.858006 12893 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.129:42365
I20260929 02:17:16.858047 13019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.129:42365 every 8 connection(s)
I20260929 02:17:16.859325 12893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/data/info.pb
I20260929 02:17:16.864537 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 12893
I20260929 02:17:16.864706 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/wal/instance
I20260929 02:17:16.865234 25442 external_mini_cluster.cc:1428] Setting key 28707a1ba2d4d12b704a3049897d3258
I20260929 02:17:16.867923 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.130:0
--local_ip_for_outbound_sockets=127.24.216.130
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35717
--builtin_ntp_servers=127.24.216.148:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:17:16.873749 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:16.862642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:35717 (local address 127.24.216.129:51011)
0929 02:17:16.863501 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:16.863540 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:16.865630 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:16.866457 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:16.866465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:16.866570 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:16.867254 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:16.867310 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:16.868461 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:16.868465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:16.869669 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:16.869682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:16.870476 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:16.870486 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:16.870723 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:16.871445 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:17:16.871447 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:16.871483 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:17:16.871553 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:16.872024 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:16.872037 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:16.872043 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 02:17:16.872263 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 02:17:16.872712 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":270,"threads_started":1}
I20260929 02:17:16.874467 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:16.861902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:51011 (local address 127.24.216.190:35717)
0929 02:17:16.862417 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:16.862424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:16.863663 (+ 1239us) server_negotiation.cc:407] Connection header received
0929 02:17:16.865953 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:16.865961 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:16.866090 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:16.866277 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:16.867457 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:16.868272 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:16.869825 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:16.870265 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:16.870393 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:16.871787 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:16.871796 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:16.871802 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:16.871823 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:16.871892 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:16.871910 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:16.871995 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:16.874103 (+ 2108us) server_negotiation.cc:299] Negotiation successful
0929 02:17:16.874242 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":274,"threads_started":1}
I20260929 02:17:16.875694 13020 heartbeater.cc:347] Connected to a master server at 127.24.216.190:35717
I20260929 02:17:16.875970 13020 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:16.876690 13020 heartbeater.cc:511] Master 127.24.216.190:35717 requested a full tablet report, sending...
I20260929 02:17:16.878995 12834 ts_manager.cc:194] Registered new tserver with Master: da51a9f512f548c0a5a425341bf27d94 (127.24.216.129:42365)
I20260929 02:17:16.880409 12834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.216.129:51011
W20260929 02:17:17.198765 13024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:17.199034 13024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:17.199172 13024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:17.206130 13024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:17.206235 13024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:17.206434 13024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.130
I20260929 02:17:17.216786 13024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.216.130
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.130
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:17.218716 13024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:17.221230 13024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:17.238116 13029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:17.238231 13030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:17.238780 13032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:17.239305 13024 server_base.cc:1034] running on GCE node
I20260929 02:17:17.240031 13024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:17.241075 13024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:17.242296 13024 hybrid_clock.cc:648] HybridClock initialized: now 1790648237242196 us; error 93 us; skew 500 ppm
I20260929 02:17:17.242725 13024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:17:17.245133 13024 webserver.cc:460] Webserver started at http://127.24.216.130:35743/ using document root <none> and password file <none>
I20260929 02:17:17.245816 13024 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:17.245939 13024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:17.246317 13024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:17.249137 13024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/data/instance:
uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8"
format_stamp: "Formatted at 2026-09-29 02:17:17 on dist-test-slave-9k0h"
server_key: "ac0697f9a0803759ca3c4be167304513"
server_key_iv: "a3a8e039a871d25e336ea558ac00ba16"
server_key_version: "encryptionkey@0"
I20260929 02:17:17.249914 13024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance:
uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8"
format_stamp: "Formatted at 2026-09-29 02:17:17 on dist-test-slave-9k0h"
server_key: "ac0697f9a0803759ca3c4be167304513"
server_key_iv: "a3a8e039a871d25e336ea558ac00ba16"
server_key_version: "encryptionkey@0"
I20260929 02:17:17.254808 13024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 02:17:17.258015 13038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:17.259306 13024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:17:17.259474 13024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/wal
uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8"
format_stamp: "Formatted at 2026-09-29 02:17:17 on dist-test-slave-9k0h"
server_key: "ac0697f9a0803759ca3c4be167304513"
server_key_iv: "a3a8e039a871d25e336ea558ac00ba16"
server_key_version: "encryptionkey@0"
I20260929 02:17:17.259697 13024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:17.273618 13024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:17.274461 13024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:17.274756 13024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:17.275645 13024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:17.276996 13024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:17.277132 13024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:17.277271 13024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:17.277365 13024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:17.311053 13024 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.130:38223
I20260929 02:17:17.311102 13150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.130:38223 every 8 connection(s)
I20260929 02:17:17.312495 13024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/data/info.pb
I20260929 02:17:17.317690 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 13024
I20260929 02:17:17.317873 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/wal/instance
I20260929 02:17:17.318411 25442 external_mini_cluster.cc:1428] Setting key 862cbdd38aaa1d73e01661cb4d1a6f39
I20260929 02:17:17.321409 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.216.131:0
--local_ip_for_outbound_sockets=127.24.216.131
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35717
--builtin_ntp_servers=127.24.216.148:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:17:17.326674 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:17.315059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.130:52733 (local address 127.24.216.190:35717)
0929 02:17:17.315238 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:17.315242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:17.316503 (+ 1261us) server_negotiation.cc:407] Connection header received
0929 02:17:17.318369 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:17.318375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:17.318471 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:17.318590 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:17.319719 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.320846 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.323378 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.324008 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.324127 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:17.325751 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:17.325757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:17.325762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:17.325784 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:17.325820 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:17.325836 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:17.325933 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:17.326237 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 02:17:17.326328 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":56}
I20260929 02:17:17.327932 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:17.315498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:35717 (local address 127.24.216.130:52733)
0929 02:17:17.316363 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:17.316397 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:17.318040 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:17.318798 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:17.318806 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:17.318887 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:17.319557 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:17.319576 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.321027 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.321035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:17.323223 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:17.323241 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.324240 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.324249 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:17.324559 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:17.325401 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:17:17.325406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:17.325456 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:17:17.325553 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:17.325964 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:17.325977 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:17.325987 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:17:17.326199 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 02:17:17.326597 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":259,"threads_started":1}
I20260929 02:17:17.329704 13151 heartbeater.cc:347] Connected to a master server at 127.24.216.190:35717
I20260929 02:17:17.330021 13151 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:17.330977 13151 heartbeater.cc:511] Master 127.24.216.190:35717 requested a full tablet report, sending...
I20260929 02:17:17.332531 12834 ts_manager.cc:194] Registered new tserver with Master: d04ed3c3d2ba4678ba5e6e509b013bc8 (127.24.216.130:38223)
I20260929 02:17:17.333614 12834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.216.130:52733
W20260929 02:17:17.659286 13154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:17.659560 13154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:17.659749 13154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:17.667411 13154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:17.667518 13154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:17.667723 13154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.216.131
I20260929 02:17:17.676723 13154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.216.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.216.131
--webserver_port=0
--tserver_master_addrs=127.24.216.190:35717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.216.131
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:17.677995 13154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:17.679787 13154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:17.692881 13159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:17.692876 13160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:17.692909 13162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:17.693574 13154 server_base.cc:1034] running on GCE node
I20260929 02:17:17.694244 13154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:17.695113 13154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:17.696319 13154 hybrid_clock.cc:648] HybridClock initialized: now 1790648237696284 us; error 28 us; skew 500 ppm
I20260929 02:17:17.696638 13154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:17:17.698846 13154 webserver.cc:460] Webserver started at http://127.24.216.131:46211/ using document root <none> and password file <none>
I20260929 02:17:17.699525 13154 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:17.699653 13154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:17.699981 13154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:17.702824 13154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/data/instance:
uuid: "5a164e667b324e81a207c321f34604a9"
format_stamp: "Formatted at 2026-09-29 02:17:17 on dist-test-slave-9k0h"
server_key: "74d705980e767c22f82d4cc9acce2e34"
server_key_iv: "3d4a53512057747859d138f184e42a2b"
server_key_version: "encryptionkey@0"
I20260929 02:17:17.703605 13154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance:
uuid: "5a164e667b324e81a207c321f34604a9"
format_stamp: "Formatted at 2026-09-29 02:17:17 on dist-test-slave-9k0h"
server_key: "74d705980e767c22f82d4cc9acce2e34"
server_key_iv: "3d4a53512057747859d138f184e42a2b"
server_key_version: "encryptionkey@0"
I20260929 02:17:17.708526 13154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:17:17.711828 13168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:17.713156 13154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:17:17.713320 13154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/wal
uuid: "5a164e667b324e81a207c321f34604a9"
format_stamp: "Formatted at 2026-09-29 02:17:17 on dist-test-slave-9k0h"
server_key: "74d705980e767c22f82d4cc9acce2e34"
server_key_iv: "3d4a53512057747859d138f184e42a2b"
server_key_version: "encryptionkey@0"
I20260929 02:17:17.713521 13154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:17.748984 13154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:17.749825 13154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:17.750128 13154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:17.751091 13154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:17:17.752490 13154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:17:17.752619 13154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:17.752749 13154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:17:17.752837 13154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:17.785209 13154 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.131:38391
I20260929 02:17:17.785295 13280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.131:38391 every 8 connection(s)
I20260929 02:17:17.786569 13154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/data/info.pb
I20260929 02:17:17.791249 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 13154
I20260929 02:17:17.791419 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/wal/instance
I20260929 02:17:17.791976 25442 external_mini_cluster.cc:1428] Setting key 5efd2fb2245c5608d20766e386e4041e
I20260929 02:17:17.799353 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:17.788986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:60229 (local address 127.24.216.190:35717)
0929 02:17:17.789200 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:17.789206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:17.790707 (+ 1501us) server_negotiation.cc:407] Connection header received
0929 02:17:17.793092 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:17.793100 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:17.793215 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:17.793334 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:17.794592 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.795433 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.796873 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.797260 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.797352 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:17.798709 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:17.798714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:17.798717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:17.798733 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:17.798766 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:17.798779 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:17.798851 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:17.799051 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 02:17:17.799123 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:17:17.800268 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:17.789587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:35717 (local address 127.24.216.131:60229)
0929 02:17:17.790566 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:17.790606 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:17.792817 (+ 2211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:17.793592 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:17.793603 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:17.793719 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:17.794425 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:17.794443 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.795572 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.795575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:17.796757 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:17.796768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.797438 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.797442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:17.797625 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:17.798385 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:17:17.798388 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:17.798436 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:17:17.798539 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:17.798862 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:17.798869 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:17.798875 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 02:17:17.799011 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 02:17:17.799273 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":260,"threads_started":1}
I20260929 02:17:17.801466 13281 heartbeater.cc:347] Connected to a master server at 127.24.216.190:35717
I20260929 02:17:17.801702 13281 heartbeater.cc:464] Registering TS with master...
I20260929 02:17:17.802381 13281 heartbeater.cc:511] Master 127.24.216.190:35717 requested a full tablet report, sending...
I20260929 02:17:17.803651 12834 ts_manager.cc:194] Registered new tserver with Master: 5a164e667b324e81a207c321f34604a9 (127.24.216.131:38391)
I20260929 02:17:17.804450 12834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.216.131:60229
I20260929 02:17:17.806951 25442 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:17:17.818878 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:17.811519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.24.216.190:35717)
0929 02:17:17.811695 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:17.811699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:17.811863 (+ 164us) server_negotiation.cc:407] Connection header received
0929 02:17:17.812133 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:17.812139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:17.812235 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:17.812336 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:17.813014 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.814255 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.815729 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.816392 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.816536 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:17.817971 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:17.817980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:17.817985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:17.818006 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:17.818045 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:17.818074 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:17.818201 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:17.818436 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 02:17:17.818538 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:17:17.822497 12834 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:17:17.829098 12834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:17:17.856899 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:17.840388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:42365 (local address 127.0.0.1:57018)
0929 02:17:17.846795 (+ 6407us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:17.846829 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:17.847037 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:17.849268 (+ 2231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:17.849275 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:17.849303 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:17.849671 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:17.849694 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.851657 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.851662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:17.854024 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:17.854040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.855921 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.855928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:17.856111 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:17.856179 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 02:17:17.856339 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 02:17:17.856537 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6170,"thread_start_us":6033,"threads_started":1}
I20260929 02:17:17.856917 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:17.840578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.24.216.129:42365)
0929 02:17:17.841656 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:17.841668 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:17.848553 (+ 6885us) server_negotiation.cc:407] Connection header received
0929 02:17:17.848689 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:17.848700 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:17.849009 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:17.849263 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:17.849896 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.851489 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.854274 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.855730 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.855885 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:17.856101 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:17.856389 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 02:17:17.856551 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":766,"threads_started":1}
I20260929 02:17:17.858201 13295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:17.841069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.24.216.130:38223)
0929 02:17:17.849866 (+ 8797us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:17.849881 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:17.849907 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:17:17.850002 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:17.850012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:17.850279 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:17.851062 (+ 783us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:17.851279 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.852709 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.855206 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.856606 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.856770 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:17.857446 (+ 676us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:17.857691 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 02:17:17.857868 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8617,"thread_start_us":8526,"threads_started":1}
I20260929 02:17:17.858328 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:17.840867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:38223 (local address 127.0.0.1:54518)
0929 02:17:17.848157 (+ 7290us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:17.848192 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:17.848377 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:17.850487 (+ 2110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:17.850492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:17.850520 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:17.850884 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:17.850895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.852861 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.852869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:17.854995 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:17.855009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.857270 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.857280 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:17.857456 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:17.857495 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:17:17.857586 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 02:17:17.857994 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7145,"thread_start_us":11970,"threads_started":1}
I20260929 02:17:17.861121 12955 tablet_service.cc:1467] Processing CreateTablet for tablet 870a3887bbb04145a665e8af9a3c4816 (DEFAULT_TABLE table=test-table [id=5ad189ec8aac4541831b8a051b474972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:17.861729 13086 tablet_service.cc:1467] Processing CreateTablet for tablet 870a3887bbb04145a665e8af9a3c4816 (DEFAULT_TABLE table=test-table [id=5ad189ec8aac4541831b8a051b474972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:17.862829 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870a3887bbb04145a665e8af9a3c4816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:17.863035 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870a3887bbb04145a665e8af9a3c4816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:17.864789 13296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:17.841408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.24.216.131:38391)
0929 02:17:17.853490 (+ 12082us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:17.853499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:17.853532 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:17:17.853615 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:17.853621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:17.853810 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:17.853992 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:17.854957 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.856089 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.860612 (+ 4523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.861965 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.862197 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:17.862394 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:17.864232 (+ 1838us) server_negotiation.cc:299] Negotiation successful
0929 02:17:17.864399 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11913,"thread_start_us":11764,"threads_started":1}
I20260929 02:17:17.865087 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:17.842063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:38391 (local address 127.0.0.1:56988)
0929 02:17:17.851583 (+ 9520us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:17.851606 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:17.851745 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:17.854297 (+ 2552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:17.854302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:17.854331 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:17.854750 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:17.854762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.856232 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.856239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:17.860396 (+ 4157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:17.860411 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:17.862518 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:17.862524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:17.862684 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:17.862726 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 02:17:17.862808 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 02:17:17.862937 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9370,"thread_start_us":5828,"threads_started":1}
I20260929 02:17:17.868546 13216 tablet_service.cc:1467] Processing CreateTablet for tablet 870a3887bbb04145a665e8af9a3c4816 (DEFAULT_TABLE table=test-table [id=5ad189ec8aac4541831b8a051b474972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:17:17.869932 13216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870a3887bbb04145a665e8af9a3c4816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:17.875568 13299 tablet_bootstrap.cc:492] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94: Bootstrap starting.
I20260929 02:17:17.879104 13299 tablet_bootstrap.cc:654] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:17.880112 13299 log.cc:824] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:17.881057 13300 tablet_bootstrap.cc:492] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8: Bootstrap starting.
I20260929 02:17:17.883996 13020 heartbeater.cc:503] Master 127.24.216.190:35717 was elected leader, sending a full tablet report...
I20260929 02:17:17.884169 13302 tablet_bootstrap.cc:492] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9: Bootstrap starting.
I20260929 02:17:17.884485 13299 tablet_bootstrap.cc:492] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94: No bootstrap required, opened a new log
I20260929 02:17:17.884725 13300 tablet_bootstrap.cc:654] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:17.884815 13299 ts_tablet_manager.cc:1397] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:17:17.887344 13300 log.cc:824] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:17.888360 13302 tablet_bootstrap.cc:654] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:17.889694 13302 log.cc:824] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:17.892431 13300 tablet_bootstrap.cc:492] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8: No bootstrap required, opened a new log
I20260929 02:17:17.892669 13302 tablet_bootstrap.cc:492] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9: No bootstrap required, opened a new log
I20260929 02:17:17.892707 13300 ts_tablet_manager.cc:1397] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260929 02:17:17.892915 13302 ts_tablet_manager.cc:1397] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 02:17:17.896113 13299 raft_consensus.cc:348] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da51a9f512f548c0a5a425341bf27d94" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42365 } } peers { permanent_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38223 } } peers { permanent_uuid: "5a164e667b324e81a207c321f34604a9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38391 } }
I20260929 02:17:17.896687 13299 raft_consensus.cc:374] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:17.896845 13299 raft_consensus.cc:729] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da51a9f512f548c0a5a425341bf27d94, State: Initialized, Role: FOLLOWER
I20260929 02:17:17.897534 13299 consensus_queue.cc:260] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da51a9f512f548c0a5a425341bf27d94" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42365 } } peers { permanent_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38223 } } peers { permanent_uuid: "5a164e667b324e81a207c321f34604a9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38391 } }
I20260929 02:17:17.899282 13299 ts_tablet_manager.cc:1428] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260929 02:17:17.901147 13300 raft_consensus.cc:348] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da51a9f512f548c0a5a425341bf27d94" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42365 } } peers { permanent_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38223 } } peers { permanent_uuid: "5a164e667b324e81a207c321f34604a9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38391 } }
I20260929 02:17:17.901582 13300 raft_consensus.cc:374] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:17.901698 13300 raft_consensus.cc:729] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04ed3c3d2ba4678ba5e6e509b013bc8, State: Initialized, Role: FOLLOWER
I20260929 02:17:17.902372 13300 consensus_queue.cc:260] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da51a9f512f548c0a5a425341bf27d94" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42365 } } peers { permanent_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38223 } } peers { permanent_uuid: "5a164e667b324e81a207c321f34604a9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38391 } }
I20260929 02:17:17.903023 13302 raft_consensus.cc:348] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da51a9f512f548c0a5a425341bf27d94" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42365 } } peers { permanent_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38223 } } peers { permanent_uuid: "5a164e667b324e81a207c321f34604a9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38391 } }
I20260929 02:17:17.903425 13151 heartbeater.cc:503] Master 127.24.216.190:35717 was elected leader, sending a full tablet report...
I20260929 02:17:17.903446 13302 raft_consensus.cc:374] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:17.903537 13302 raft_consensus.cc:729] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a164e667b324e81a207c321f34604a9, State: Initialized, Role: FOLLOWER
I20260929 02:17:17.903968 13300 ts_tablet_manager.cc:1428] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 02:17:17.904078 13302 consensus_queue.cc:260] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da51a9f512f548c0a5a425341bf27d94" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42365 } } peers { permanent_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38223 } } peers { permanent_uuid: "5a164e667b324e81a207c321f34604a9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38391 } }
I20260929 02:17:17.905107 13281 heartbeater.cc:503] Master 127.24.216.190:35717 was elected leader, sending a full tablet report...
I20260929 02:17:17.905557 13302 ts_tablet_manager.cc:1428] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260929 02:17:18.038205 13282 tablet.cc:2369] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:17:18.064531 13152 tablet.cc:2369] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:17:18.111889 13021 tablet.cc:2369] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:17:18.303834 13305 raft_consensus.cc:482] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:17:18.304153 13305 raft_consensus.cc:504] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da51a9f512f548c0a5a425341bf27d94" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42365 } } peers { permanent_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38223 } } peers { permanent_uuid: "5a164e667b324e81a207c321f34604a9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38391 } }
I20260929 02:17:18.304340 13307 raft_consensus.cc:482] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:17:18.304589 13307 raft_consensus.cc:504] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da51a9f512f548c0a5a425341bf27d94" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42365 } } peers { permanent_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38223 } } peers { permanent_uuid: "5a164e667b324e81a207c321f34604a9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38391 } }
I20260929 02:17:18.305948 13305 leader_election.cc:288] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d04ed3c3d2ba4678ba5e6e509b013bc8 (127.24.216.130:38223), 5a164e667b324e81a207c321f34604a9 (127.24.216.131:38391)
I20260929 02:17:18.311292 13307 leader_election.cc:288] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da51a9f512f548c0a5a425341bf27d94 (127.24.216.129:42365), d04ed3c3d2ba4678ba5e6e509b013bc8 (127.24.216.130:38223)
I20260929 02:17:18.326349 13296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:18.318921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:37931 (local address 127.24.216.131:38391)
0929 02:17:18.319087 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:18.319093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:18.319110 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:17:18.319286 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:18.319290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:18.319351 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:18.319414 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:18.320359 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.321507 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.324132 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.325275 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.325397 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:18.325464 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:18.325953 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 02:17:18.326077 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 02:17:18.327376 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:18.306454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.131:38391 (local address 127.24.216.129:37931)
0929 02:17:18.318664 (+ 12210us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:18.318991 (+ 327us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:18.319113 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:18.319663 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:18.319669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:18.319698 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:18.320135 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:18.320148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.321674 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.321679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:18.323926 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:18.323940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.325620 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.325626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:18.325789 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:18.325820 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:17:18.326518 (+ 698us) client_negotiation.cc:240] Negotiation successful
0929 02:17:18.326885 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12068,"mutex_wait_us":164,"thread_start_us":11380,"threads_started":1}
I20260929 02:17:18.328490 13295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:18.306604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.129:50947 (local address 127.24.216.130:38223)
0929 02:17:18.306870 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:18.306881 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:18.318668 (+ 11787us) server_negotiation.cc:407] Connection header received
0929 02:17:18.319596 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:18.319602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:18.319687 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:18.319759 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:18.320506 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.321829 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.324583 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.325814 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.325925 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:18.325985 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:18.328094 (+ 2109us) server_negotiation.cc:299] Negotiation successful
0929 02:17:18.328242 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 02:17:18.328480 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870a3887bbb04145a665e8af9a3c4816" candidate_uuid: "da51a9f512f548c0a5a425341bf27d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a164e667b324e81a207c321f34604a9" is_pre_election: true
I20260929 02:17:18.328982 13236 raft_consensus.cc:2454] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da51a9f512f548c0a5a425341bf27d94 in term 0.
I20260929 02:17:18.329133 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:18.306115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:38223 (local address 127.24.216.129:50947)
0929 02:17:18.318566 (+ 12451us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:18.318586 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:18.319436 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:18.319923 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:18.319928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:18.319950 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:18.320364 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:18.320374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.322058 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.322066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:18.324396 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:18.324410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.326134 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.326139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:18.326424 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:18.326610 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 02:17:18.328653 (+ 2043us) client_negotiation.cc:240] Negotiation successful
0929 02:17:18.328796 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11911,"mutex_wait_us":1,"thread_start_us":12239,"threads_started":1}
I20260929 02:17:18.329774 12910 leader_election.cc:302] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a164e667b324e81a207c321f34604a9, da51a9f512f548c0a5a425341bf27d94; no voters:
I20260929 02:17:18.330281 13305 raft_consensus.cc:2781] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:17:18.330127 13106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870a3887bbb04145a665e8af9a3c4816" candidate_uuid: "da51a9f512f548c0a5a425341bf27d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8" is_pre_election: true
I20260929 02:17:18.330420 13305 raft_consensus.cc:482] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:17:18.330523 13305 raft_consensus.cc:3037] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:18.330713 13106 raft_consensus.cc:2454] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da51a9f512f548c0a5a425341bf27d94 in term 0.
I20260929 02:17:18.333685 13305 raft_consensus.cc:504] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da51a9f512f548c0a5a425341bf27d94" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42365 } } peers { permanent_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38223 } } peers { permanent_uuid: "5a164e667b324e81a207c321f34604a9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38391 } }
I20260929 02:17:18.334705 13305 leader_election.cc:288] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [CANDIDATE]: Term 1 election: Requested vote from peers d04ed3c3d2ba4678ba5e6e509b013bc8 (127.24.216.130:38223), 5a164e667b324e81a207c321f34604a9 (127.24.216.131:38391)
I20260929 02:17:18.334705 13106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870a3887bbb04145a665e8af9a3c4816" candidate_uuid: "da51a9f512f548c0a5a425341bf27d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8"
I20260929 02:17:18.334802 13236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870a3887bbb04145a665e8af9a3c4816" candidate_uuid: "da51a9f512f548c0a5a425341bf27d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a164e667b324e81a207c321f34604a9"
I20260929 02:17:18.334905 13106 raft_consensus.cc:3037] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:18.334993 13236 raft_consensus.cc:3037] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:18.338013 13311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:18.306903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:41347 (local address 127.24.216.130:38223)
0929 02:17:18.315959 (+ 9056us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:18.315967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:18.318991 (+ 3024us) server_negotiation.cc:407] Connection header received
0929 02:17:18.319275 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:18.319281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:18.319351 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:18.319417 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:18.326691 (+ 7274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.327839 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.335096 (+ 7257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.336529 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.336684 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:18.336757 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:18.337296 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 02:17:18.337453 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8878,"thread_start_us":8636,"threads_started":1}
I20260929 02:17:18.338085 13236 raft_consensus.cc:2454] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da51a9f512f548c0a5a425341bf27d94 in term 1.
I20260929 02:17:18.338683 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:18.306115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.130:38223 (local address 127.24.216.131:41347)
0929 02:17:18.318889 (+ 12774us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:18.318909 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:18.319113 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:18.319832 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:18.319838 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:18.319865 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:18.320138 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:18.320147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.328010 (+ 7863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.328015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:18.334895 (+ 6880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:18.334911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.336897 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.336905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:18.337090 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:18.337135 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 02:17:18.338194 (+ 1059us) client_negotiation.cc:240] Negotiation successful
0929 02:17:18.338348 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12635,"thread_start_us":12515,"threads_started":1}
I20260929 02:17:18.338620 13106 raft_consensus.cc:2454] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da51a9f512f548c0a5a425341bf27d94 in term 1.
I20260929 02:17:18.338649 12910 leader_election.cc:302] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a164e667b324e81a207c321f34604a9, da51a9f512f548c0a5a425341bf27d94; no voters:
I20260929 02:17:18.339118 13305 raft_consensus.cc:2781] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:18.339550 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870a3887bbb04145a665e8af9a3c4816" candidate_uuid: "5a164e667b324e81a207c321f34604a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8" is_pre_election: true
I20260929 02:17:18.340049 13105 raft_consensus.cc:2379] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a164e667b324e81a207c321f34604a9 in current term 1: Already voted for candidate da51a9f512f548c0a5a425341bf27d94 in this term.
I20260929 02:17:18.341078 13305 raft_consensus.cc:686] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [term 1 LEADER]: Becoming Leader. State: Replica: da51a9f512f548c0a5a425341bf27d94, State: Running, Role: LEADER
I20260929 02:17:18.341588 13305 consensus_queue.cc:237] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da51a9f512f548c0a5a425341bf27d94" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42365 } } peers { permanent_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38223 } } peers { permanent_uuid: "5a164e667b324e81a207c321f34604a9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38391 } }
I20260929 02:17:18.345592 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:18.306115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.129:42365 (local address 127.24.216.131:42779)
0929 02:17:18.338926 (+ 32811us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:18.338949 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:18.339121 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:18.339657 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:18.339662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:18.339686 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:17:18.339837 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:18.339846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.341215 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.341219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:18.343002 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:18.343017 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.344855 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.344860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:18.345037 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:18.345066 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:17:18.345151 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 02:17:18.345285 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32692,"mutex_wait_us":152,"thread_start_us":32787,"threads_started":1}
I20260929 02:17:18.345894 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:18.307417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.216.131:42779 (local address 127.24.216.129:42365)
0929 02:17:18.307635 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:18.307644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:18.339021 (+ 31377us) server_negotiation.cc:407] Connection header received
0929 02:17:18.339292 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:18.339301 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:18.339398 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:18.339462 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:17:18.339992 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.341030 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.343175 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.344500 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.344673 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:18.344754 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:18.345544 (+ 790us) server_negotiation.cc:299] Negotiation successful
0929 02:17:18.345667 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 02:17:18.346732 12975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870a3887bbb04145a665e8af9a3c4816" candidate_uuid: "5a164e667b324e81a207c321f34604a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da51a9f512f548c0a5a425341bf27d94" is_pre_election: true
I20260929 02:17:18.346619 12833 catalog_manager.cc:5495] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 reported cstate change: term changed from 0 to 1, leader changed from <none> to da51a9f512f548c0a5a425341bf27d94 (127.24.216.129). New cstate: current_term: 1 leader_uuid: "da51a9f512f548c0a5a425341bf27d94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da51a9f512f548c0a5a425341bf27d94" member_type: VOTER last_known_addr { host: "127.24.216.129" port: 42365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a164e667b324e81a207c321f34604a9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260929 02:17:18.347728 13172 leader_election.cc:302] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a164e667b324e81a207c321f34604a9; no voters: d04ed3c3d2ba4678ba5e6e509b013bc8, da51a9f512f548c0a5a425341bf27d94
I20260929 02:17:18.348134 13307 raft_consensus.cc:2726] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:17:18.420431 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:18.414879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.24.216.129:42365)
0929 02:17:18.415115 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:18.415121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:18.415424 (+ 303us) server_negotiation.cc:407] Connection header received
0929 02:17:18.415591 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:18.415594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:18.415668 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:18.415795 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:18.416174 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.416964 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.418448 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.418850 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.418943 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:18.419339 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:18.419857 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:18.419961 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:18.420047 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 02:17:18.420108 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:17:18.428076 13105 raft_consensus.cc:1261] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8 [term 1 FOLLOWER]: Refusing update from remote peer da51a9f512f548c0a5a425341bf27d94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:17:18.428100 13236 raft_consensus.cc:1261] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [term 1 FOLLOWER]: Refusing update from remote peer da51a9f512f548c0a5a425341bf27d94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:17:18.429039 13313 consensus_queue.cc:1035] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a164e667b324e81a207c321f34604a9" member_type: VOTER last_known_addr { host: "127.24.216.131" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:17:18.429383 13305 consensus_queue.cc:1035] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d04ed3c3d2ba4678ba5e6e509b013bc8" member_type: VOTER last_known_addr { host: "127.24.216.130" port: 38223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:17:18.453096 12834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:17:18.453367 12834 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:17:18.456686 12834 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c76e51c85a8c4d1cb84f35bccd6db766: Sending DeleteTablet for 3 replicas of tablet 870a3887bbb04145a665e8af9a3c4816
I20260929 02:17:18.457720 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet 870a3887bbb04145a665e8af9a3c4816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:18 UTC) from {username='slave'} at 127.0.0.1:57018
I20260929 02:17:18.457775 13086 tablet_service.cc:1514] Processing DeleteTablet for tablet 870a3887bbb04145a665e8af9a3c4816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:18 UTC) from {username='slave'} at 127.0.0.1:54518
I20260929 02:17:18.458042 13216 tablet_service.cc:1514] Processing DeleteTablet for tablet 870a3887bbb04145a665e8af9a3c4816 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:17:18 UTC) from {username='slave'} at 127.0.0.1:56988
I20260929 02:17:18.459640 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 12893
I20260929 02:17:18.470077 13331 tablet_replica.cc:330] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94: stopping tablet replica
I20260929 02:17:18.470290 13332 tablet_replica.cc:330] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8: stopping tablet replica
I20260929 02:17:18.470858 13331 raft_consensus.cc:2229] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:17:18.470861 13332 raft_consensus.cc:2229] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:17:18.471527 13332 raft_consensus.cc:2258] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:18.471823 13331 raft_consensus.cc:2258] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:18.473783 13331 ts_tablet_manager.cc:1905] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:18.474313 13332 ts_tablet_manager.cc:1905] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:18.481945 13331 ts_tablet_manager.cc:1918] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:18.482218 13331 log.cc:1196] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-0/wal/wals/870a3887bbb04145a665e8af9a3c4816
I20260929 02:17:18.482836 13331 ts_tablet_manager.cc:1939] T 870a3887bbb04145a665e8af9a3c4816 P da51a9f512f548c0a5a425341bf27d94: Deleting consensus metadata
I20260929 02:17:18.483767 13332 ts_tablet_manager.cc:1918] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:18.483956 13332 log.cc:1196] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-1/wal/wals/870a3887bbb04145a665e8af9a3c4816
I20260929 02:17:18.484552 13332 ts_tablet_manager.cc:1939] T 870a3887bbb04145a665e8af9a3c4816 P d04ed3c3d2ba4678ba5e6e509b013bc8: Deleting consensus metadata
I20260929 02:17:18.484834 12821 catalog_manager.cc:4841] TS da51a9f512f548c0a5a425341bf27d94 (127.24.216.129:42365): tablet 870a3887bbb04145a665e8af9a3c4816 (table test-table [id=5ad189ec8aac4541831b8a051b474972]) successfully deleted
I20260929 02:17:18.486620 12819 catalog_manager.cc:4841] TS d04ed3c3d2ba4678ba5e6e509b013bc8 (127.24.216.130:38223): tablet 870a3887bbb04145a665e8af9a3c4816 (table test-table [id=5ad189ec8aac4541831b8a051b474972]) successfully deleted
I20260929 02:17:18.488613 13333 tablet_replica.cc:330] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9: stopping tablet replica
I20260929 02:17:18.489059 13333 raft_consensus.cc:2229] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:17:18.489564 13333 raft_consensus.cc:2258] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:17:18.490550 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:18.461186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57032 (local address 127.24.216.129:42365)
0929 02:17:18.461420 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:18.461429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:18.481519 (+ 20090us) server_negotiation.cc:407] Connection header received
0929 02:17:18.482005 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:18.482012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:18.482135 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:18.482265 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:18.482967 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.484042 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.486540 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.487237 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.487346 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:18.488942 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:18.488951 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:18.488964 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:18.489626 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:18.489706 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:18.489736 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:18.489849 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:18.490033 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 02:17:18.490139 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:17:18.491321 13015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:18.491711 13333 ts_tablet_manager.cc:1905] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:17:18.497645 13333 ts_tablet_manager.cc:1918] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:17:18.497808 13333 log.cc:1196] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9: Deleting WAL directory at /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790647920678408-25442-0/minicluster-data/ts-2/wal/wals/870a3887bbb04145a665e8af9a3c4816
I20260929 02:17:18.498308 13333 ts_tablet_manager.cc:1939] T 870a3887bbb04145a665e8af9a3c4816 P 5a164e667b324e81a207c321f34604a9: Deleting consensus metadata
I20260929 02:17:18.499661 12820 catalog_manager.cc:4841] TS 5a164e667b324e81a207c321f34604a9 (127.24.216.131:38391): tablet 870a3887bbb04145a665e8af9a3c4816 (table test-table [id=5ad189ec8aac4541831b8a051b474972]) successfully deleted
I20260929 02:17:18.584750 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 12893
I20260929 02:17:18.602730 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 13024
I20260929 02:17:18.608868 13311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:18.603426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.24.216.130:38223)
0929 02:17:18.603605 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:18.603609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:18.603627 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:17:18.603748 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:18.603752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:18.603845 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:18.603939 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:18.604288 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.604967 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.606106 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.606575 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.606668 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:18.607635 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:18.607642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:18.607655 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:18.608217 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:18.608260 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:18.608279 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:18.608365 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:18.608468 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 02:17:18.608541 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:17:18.609494 13146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:18.700533 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 13024
I20260929 02:17:18.719656 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 13154
I20260929 02:17:18.726123 13296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:18.720426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.24.216.131:38391)
0929 02:17:18.720646 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:18.720650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:18.720669 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:17:18.720782 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:18.720786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:18.720886 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:18.720968 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:18.721334 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.722039 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.723282 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:18.723715 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:18.723807 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:18.724792 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:18.724800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:18.724815 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:18.725398 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:18.725445 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:18.725469 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:18.725554 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:18.725680 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 02:17:18.725765 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:17:18.726979 13276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:18.816979 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 13154
I20260929 02:17:18.835486 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 12803
I20260929 02:17:18.836642 12864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:18.944713 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 12803
2026-09-29T02:17:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3118 ms)
[----------] 6 tests from EncryptionPolicyTest (21498 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:19 dist-test-slave-9k0h krb5kdc[13341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:19 dist-test-slave-9k0h krb5kdc[13341](info): set up 2 sockets
Sep 29 02:17:19 dist-test-slave-9k0h krb5kdc[13341](info): commencing operation
krb5kdc: starting...
W20260929 02:17:20.062680 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:20 dist-test-slave-9k0h krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:17:20.319634 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:20.642450 13354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:20.642722 13354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:20.642890 13354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:20.649967 13354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:20.650116 13354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:20.650266 13354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:20.650385 13354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:20.650463 13354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:20.660099 13354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33207
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:20.661381 13354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:20.662722 13354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:20.674209 13359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:20.674120 13360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:20.674176 13362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:20.675225 13354 server_base.cc:1034] running on GCE node
I20260929 02:17:20.675930 13354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:20.677237 13354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:20.678471 13354 hybrid_clock.cc:648] HybridClock initialized: now 1790648240678477 us; error 95 us; skew 500 ppm
I20260929 02:17:20.678809 13354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:20 dist-test-slave-9k0h krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:20.688470 13354 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:17:20.690100 13354 webserver.cc:460] Webserver started at http://127.0.0.1:34807/ using document root <none> and password file <none>
I20260929 02:17:20.690851 13354 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:20.690997 13354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:20.691298 13354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:20.694742 13354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "c384183881f0490e9376395c656c0235"
format_stamp: "Formatted at 2026-09-29 02:17:20 on dist-test-slave-9k0h"
server_key: "441de4f256672ff56e421edc2ae59dfd"
server_key_iv: "e84a83275f1a993659e04d7775651b92"
server_key_version: "encryptionkey@0"
I20260929 02:17:20.695487 13354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "c384183881f0490e9376395c656c0235"
format_stamp: "Formatted at 2026-09-29 02:17:20 on dist-test-slave-9k0h"
server_key: "441de4f256672ff56e421edc2ae59dfd"
server_key_iv: "e84a83275f1a993659e04d7775651b92"
server_key_version: "encryptionkey@0"
I20260929 02:17:20.700556 13354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:17:20.703852 13369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:20.705194 13354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:17:20.705368 13354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "c384183881f0490e9376395c656c0235"
format_stamp: "Formatted at 2026-09-29 02:17:20 on dist-test-slave-9k0h"
server_key: "441de4f256672ff56e421edc2ae59dfd"
server_key_iv: "e84a83275f1a993659e04d7775651b92"
server_key_version: "encryptionkey@0"
I20260929 02:17:20.705557 13354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:20.723429 13354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:20.727605 13354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:20.727898 13354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:20.746114 13354 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33207
I20260929 02:17:20.746165 13420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33207 every 8 connection(s)
I20260929 02:17:20.747503 13354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:20.750973 13421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:20.753149 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 13354
I20260929 02:17:20.753367 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:20.753981 25442 external_mini_cluster.cc:1428] Setting key 6e37ced87c4d05df446834f600cfb7d7
I20260929 02:17:20.763520 13421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235: Bootstrap starting.
I20260929 02:17:20.766945 13421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:20.767895 13421 log.cc:824] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:17:20 dist-test-slave-9k0h krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:17:20.770584 13421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235: No bootstrap required, opened a new log
I20260929 02:17:20.777079 13424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:20.756031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58245 (local address 127.0.0.1:33207)
0929 02:17:20.756820 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:20.756835 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:20.756880 (+ 45us) server_negotiation.cc:407] Connection header received
0929 02:17:20.759160 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:20.759189 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:20.759677 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:20.760112 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:20.761428 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:20.763358 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:20.764637 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:20.765713 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:20.765895 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:20.769911 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:20.769918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:20.769931 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:20.769962 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:20.773035 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:20.773808 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:20.773814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:20.773822 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:20.773985 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:20.774511 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:20.774514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:20.774516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:20.775001 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:20.775352 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:20.775855 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 02:17:20.776152 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":352,"threads_started":1}
I20260929 02:17:20.778410 13421 raft_consensus.cc:348] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c384183881f0490e9376395c656c0235" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33207 } }
I20260929 02:17:20.778848 13421 raft_consensus.cc:374] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:20.778947 13421 raft_consensus.cc:729] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c384183881f0490e9376395c656c0235, State: Initialized, Role: FOLLOWER
I20260929 02:17:20.779542 13421 consensus_queue.cc:260] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c384183881f0490e9376395c656c0235" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33207 } }
I20260929 02:17:20.779825 13421 raft_consensus.cc:388] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:20.779965 13421 raft_consensus.cc:482] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:20.780159 13421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:20.781025 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:20.782505 13421 raft_consensus.cc:504] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c384183881f0490e9376395c656c0235" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33207 } }
I20260929 02:17:20.782955 13421 leader_election.cc:302] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c384183881f0490e9376395c656c0235; no voters:
I20260929 02:17:20.783509 13421 leader_election.cc:288] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:20.783659 13426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:20.785091 13426 raft_consensus.cc:686] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [term 1 LEADER]: Becoming Leader. State: Replica: c384183881f0490e9376395c656c0235, State: Running, Role: LEADER
I20260929 02:17:20.785805 13421 sys_catalog.cc:564] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:20.785832 13426 consensus_queue.cc:237] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c384183881f0490e9376395c656c0235" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33207 } }
I20260929 02:17:20.790474 13428 sys_catalog.cc:455] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c384183881f0490e9376395c656c0235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c384183881f0490e9376395c656c0235" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33207 } } }
I20260929 02:17:20.790894 13428 sys_catalog.cc:458] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:20.791384 13430 sys_catalog.cc:455] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c384183881f0490e9376395c656c0235. Latest consensus state: current_term: 1 leader_uuid: "c384183881f0490e9376395c656c0235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c384183881f0490e9376395c656c0235" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33207 } } }
I20260929 02:17:20.791689 13430 sys_catalog.cc:458] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:20.808614 13424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:20.787404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46371 (local address 127.0.0.1:33207)
0929 02:17:20.787563 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:20.787571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:20.788950 (+ 1379us) server_negotiation.cc:407] Connection header received
0929 02:17:20.789184 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:20.789193 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:20.789306 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:20.789400 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:20.791479 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:20.792764 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:20.795042 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:20.795951 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:20.796060 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:20.799985 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:20.799995 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:20.800002 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:20.800051 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:20.804287 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:20.805443 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:20.805451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:20.805457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:20.805630 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:20.806305 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:20.806313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:20.806318 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:20.806836 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:20.807150 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:20.807928 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 02:17:20.808145 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 02:17:20.810935 13439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:20.815964 13439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:20.824785 13439 catalog_manager.cc:1347] Generated new cluster ID: 5d601effce264a2d9f071a06b4af0fe2
I20260929 02:17:20.824888 13439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:20.844439 13439 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:20.845705 13439 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:20.861640 13439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c384183881f0490e9376395c656c0235: Generated new TSK 0
I20260929 02:17:20.862545 13439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:20.868577 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 13354
I20260929 02:17:20.870096 13416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:20.961714 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 13354
W20260929 02:17:20.979172 13339 connection.cc:535] client connection to 127.0.0.1:33207 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33207): Connection reset by peer (error 104)
2026-09-29T02:17:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2006 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:21 dist-test-slave-9k0h krb5kdc[13454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:21 dist-test-slave-9k0h krb5kdc[13454](info): set up 2 sockets
Sep 29 02:17:21 dist-test-slave-9k0h krb5kdc[13454](info): commencing operation
krb5kdc: starting...
W20260929 02:17:22.078634 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:22 dist-test-slave-9k0h krb5kdc[13454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:17:22.349411 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:22.671792 13467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:22.672106 13467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:22.672242 13467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:22.679075 13467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:22.679176 13467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:22.679297 13467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:22.679376 13467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:22.679462 13467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:22.689008 13467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45361
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:22.690303 13467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:22.691578 13467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:22.703233 13472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:22.703263 13473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:22.703297 13475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:22.704051 13467 server_base.cc:1034] running on GCE node
I20260929 02:17:22.704807 13467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:22.706199 13467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:22.707433 13467 hybrid_clock.cc:648] HybridClock initialized: now 1790648242707395 us; error 34 us; skew 500 ppm
I20260929 02:17:22.707819 13467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:22 dist-test-slave-9k0h krb5kdc[13454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:22.711076 13467 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:17:22.712498 13467 webserver.cc:460] Webserver started at http://127.0.0.1:40073/ using document root <none> and password file <none>
I20260929 02:17:22.713189 13467 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:22.713337 13467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:22.713635 13467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:22.716600 13467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "3b0bb61ce740446f877dfe8c46621e75"
format_stamp: "Formatted at 2026-09-29 02:17:22 on dist-test-slave-9k0h"
server_key: "18bc43601cc88b8222bb3867675a8e1d"
server_key_iv: "af2308502ae98dc27dfaa7a97315e95b"
server_key_version: "encryptionkey@0"
I20260929 02:17:22.717376 13467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "3b0bb61ce740446f877dfe8c46621e75"
format_stamp: "Formatted at 2026-09-29 02:17:22 on dist-test-slave-9k0h"
server_key: "18bc43601cc88b8222bb3867675a8e1d"
server_key_iv: "af2308502ae98dc27dfaa7a97315e95b"
server_key_version: "encryptionkey@0"
I20260929 02:17:22.722530 13467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 02:17:22.725870 13482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:22.727213 13467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:17:22.727398 13467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "3b0bb61ce740446f877dfe8c46621e75"
format_stamp: "Formatted at 2026-09-29 02:17:22 on dist-test-slave-9k0h"
server_key: "18bc43601cc88b8222bb3867675a8e1d"
server_key_iv: "af2308502ae98dc27dfaa7a97315e95b"
server_key_version: "encryptionkey@0"
I20260929 02:17:22.727593 13467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:22.750111 13467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:22.754570 13467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:22.754863 13467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:22.772981 13467 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45361
I20260929 02:17:22.773005 13533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45361 every 8 connection(s)
I20260929 02:17:22.774356 13467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:22.777858 13534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:22.784288 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 13467
I20260929 02:17:22.784498 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:22.784909 25442 external_mini_cluster.cc:1428] Setting key 3296694a36e2a1a80891124d4d70a437
I20260929 02:17:22.788405 13534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75: Bootstrap starting.
I20260929 02:17:22.792307 13534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:22.793519 13534 log.cc:824] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:22.798336 13534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75: No bootstrap required, opened a new log
I20260929 02:17:22.802572 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:22.786703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46219 (local address 127.0.0.1:45361)
0929 02:17:22.787458 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:22.787473 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:22.787521 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:17:22.789837 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:22.789865 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:22.790418 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:22.790823 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:22.791313 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:22.793304 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:22.795625 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:22.796799 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:22.797002 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:22.799114 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:22.799123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:22.799150 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:22.799851 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:22.799949 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:22.799978 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:22.800294 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:22.800853 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 02:17:22.801037 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":316,"threads_started":1}
I20260929 02:17:22.807446 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:22.812142 13534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b0bb61ce740446f877dfe8c46621e75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45361 } }
I20260929 02:17:22.812623 13534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:22.812752 13534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b0bb61ce740446f877dfe8c46621e75, State: Initialized, Role: FOLLOWER
I20260929 02:17:22.813506 13534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b0bb61ce740446f877dfe8c46621e75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45361 } }
I20260929 02:17:22.813863 13534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:22.814020 13534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:22.814234 13534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:22.817863 13534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b0bb61ce740446f877dfe8c46621e75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45361 } }
I20260929 02:17:22.818532 13534 leader_election.cc:302] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b0bb61ce740446f877dfe8c46621e75; no voters:
I20260929 02:17:22.819206 13534 leader_election.cc:288] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:22.819394 13543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:22.822510 13543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [term 1 LEADER]: Becoming Leader. State: Replica: 3b0bb61ce740446f877dfe8c46621e75, State: Running, Role: LEADER
I20260929 02:17:22.823119 13543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b0bb61ce740446f877dfe8c46621e75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45361 } }
I20260929 02:17:22.825278 13534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:22.825928 13544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b0bb61ce740446f877dfe8c46621e75. Latest consensus state: current_term: 1 leader_uuid: "3b0bb61ce740446f877dfe8c46621e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b0bb61ce740446f877dfe8c46621e75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45361 } } }
I20260929 02:17:22.825914 13545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b0bb61ce740446f877dfe8c46621e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b0bb61ce740446f877dfe8c46621e75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45361 } } }
I20260929 02:17:22.826412 13545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:22.826412 13544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:22.827288 13548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:22.835021 13548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:22.838467 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:22.820136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52999 (local address 127.0.0.1:45361)
0929 02:17:22.820354 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:22.820362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:22.827360 (+ 6998us) server_negotiation.cc:407] Connection header received
0929 02:17:22.827705 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:22.827713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:22.827806 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:22.827889 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:22.828782 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:22.830215 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:22.833400 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:22.834258 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:22.834366 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:22.837447 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:22.837454 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:22.837459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:22.837485 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:22.837553 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:22.837577 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:22.837678 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:22.838021 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 02:17:22.838080 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:17:22.845225 13548 catalog_manager.cc:1347] Generated new cluster ID: 9c285da117ca4accbfff38df3a459be1
I20260929 02:17:22.845340 13548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:22.869777 13548 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:22.871099 13548 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:22.877229 13548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b0bb61ce740446f877dfe8c46621e75: Generated new TSK 0
I20260929 02:17:22.878003 13548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:22.888779 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 13467
W20260929 02:17:22.889932 13529 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46219
W20260929 02:17:22.890709 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 13467: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:17:22.890892 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 13467
W20260929 02:17:22.908241 13452 connection.cc:535] client connection to 127.0.0.1:45361 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45361): Connection reset by peer (error 104)
2026-09-29T02:17:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:22 dist-test-slave-9k0h krb5kdc[13565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:22 dist-test-slave-9k0h krb5kdc[13565](info): set up 2 sockets
Sep 29 02:17:22 dist-test-slave-9k0h krb5kdc[13565](info): commencing operation
krb5kdc: starting...
W20260929 02:17:23.990594 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:24 dist-test-slave-9k0h krb5kdc[13565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:17:24.253273 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:24.579294 13578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:24.579612 13578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:24.579778 13578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:24.586915 13578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:24.587055 13578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:24.587162 13578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:24.587276 13578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:24.587361 13578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:24.596987 13578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:24.598398 13578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:24.599778 13578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:24.611419 13583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:24.611510 13584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:24.611937 13586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:24.612665 13578 server_base.cc:1034] running on GCE node
I20260929 02:17:24.613427 13578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:24.614840 13578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:24.616076 13578 hybrid_clock.cc:648] HybridClock initialized: now 1790648244616042 us; error 40 us; skew 500 ppm
I20260929 02:17:24.616410 13578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:24 dist-test-slave-9k0h krb5kdc[13565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:24.619485 13578 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:17:24.620967 13578 webserver.cc:460] Webserver started at http://127.0.0.1:44457/ using document root <none> and password file <none>
I20260929 02:17:24.621615 13578 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:24.621764 13578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:24.622061 13578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:24.624894 13578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "f37aca36e15a4142bdc5b9d73690ae70"
format_stamp: "Formatted at 2026-09-29 02:17:24 on dist-test-slave-9k0h"
server_key: "3ac0fce09081bf7b7cce9068763bb85f"
server_key_iv: "4596916b6e91fa70e02e2fe4bcde0593"
server_key_version: "encryptionkey@0"
I20260929 02:17:24.625629 13578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "f37aca36e15a4142bdc5b9d73690ae70"
format_stamp: "Formatted at 2026-09-29 02:17:24 on dist-test-slave-9k0h"
server_key: "3ac0fce09081bf7b7cce9068763bb85f"
server_key_iv: "4596916b6e91fa70e02e2fe4bcde0593"
server_key_version: "encryptionkey@0"
I20260929 02:17:24.630661 13578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:17:24.634027 13593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:24.635366 13578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:17:24.635542 13578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "f37aca36e15a4142bdc5b9d73690ae70"
format_stamp: "Formatted at 2026-09-29 02:17:24 on dist-test-slave-9k0h"
server_key: "3ac0fce09081bf7b7cce9068763bb85f"
server_key_iv: "4596916b6e91fa70e02e2fe4bcde0593"
server_key_version: "encryptionkey@0"
I20260929 02:17:24.635737 13578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:24.641517 13578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:24.645236 13578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:24.645509 13578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:24.664238 13578 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34213
I20260929 02:17:24.664333 13644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34213 every 8 connection(s)
I20260929 02:17:24.665613 13578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:24.668099 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 13578
I20260929 02:17:24.668294 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:24.668720 25442 external_mini_cluster.cc:1428] Setting key 10ead6cabaab955156e4ba425c119275
I20260929 02:17:24.669565 13645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:24.677129 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:24.670561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53279 (local address 127.0.0.1:34213)
0929 02:17:24.671375 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:24.671392 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:24.671474 (+ 82us) server_negotiation.cc:407] Connection header received
0929 02:17:24.673238 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:24.673264 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:24.673797 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:24.674240 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:24.675200 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:24.675207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:24.675222 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:24.675238 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:24.675334 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:24.675357 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:24.675569 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:24.675720 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 02:17:24.675907 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":326,"threads_started":1}
I20260929 02:17:24.680533 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:24.683447 13645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70: Bootstrap starting.
I20260929 02:17:24.686964 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:24.684714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38159 (local address 127.0.0.1:34213)
0929 02:17:24.684895 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:24.684905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:24.685182 (+ 277us) server_negotiation.cc:407] Connection header received
0929 02:17:24.685307 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:24.685313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:24.685372 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:24.685473 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:24.686354 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:24.686359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:24.686363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:24.686380 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:24.686421 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:24.686439 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:24.686541 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:24.686621 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 02:17:24.686671 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:17:24.687000 13645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:24.688230 13645 log.cc:824] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:24.690737 13645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70: No bootstrap required, opened a new log
I20260929 02:17:24.697775 13645 raft_consensus.cc:348] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37aca36e15a4142bdc5b9d73690ae70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34213 } }
I20260929 02:17:24.698181 13645 raft_consensus.cc:374] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:24.698297 13645 raft_consensus.cc:729] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37aca36e15a4142bdc5b9d73690ae70, State: Initialized, Role: FOLLOWER
I20260929 02:17:24.698817 13645 consensus_queue.cc:260] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37aca36e15a4142bdc5b9d73690ae70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34213 } }
I20260929 02:17:24.699034 13645 raft_consensus.cc:388] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:24.699188 13645 raft_consensus.cc:482] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:24.699358 13645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:24.701699 13645 raft_consensus.cc:504] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37aca36e15a4142bdc5b9d73690ae70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34213 } }
I20260929 02:17:24.702272 13645 leader_election.cc:302] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f37aca36e15a4142bdc5b9d73690ae70; no voters:
I20260929 02:17:24.702814 13645 leader_election.cc:288] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:24.702943 13655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:24.703606 13655 raft_consensus.cc:686] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [term 1 LEADER]: Becoming Leader. State: Replica: f37aca36e15a4142bdc5b9d73690ae70, State: Running, Role: LEADER
I20260929 02:17:24.704135 13655 consensus_queue.cc:237] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37aca36e15a4142bdc5b9d73690ae70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34213 } }
I20260929 02:17:24.704809 13645 sys_catalog.cc:564] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:24.706769 13656 sys_catalog.cc:455] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f37aca36e15a4142bdc5b9d73690ae70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37aca36e15a4142bdc5b9d73690ae70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34213 } } }
I20260929 02:17:24.707156 13656 sys_catalog.cc:458] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:24.708437 13657 sys_catalog.cc:455] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f37aca36e15a4142bdc5b9d73690ae70. Latest consensus state: current_term: 1 leader_uuid: "f37aca36e15a4142bdc5b9d73690ae70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37aca36e15a4142bdc5b9d73690ae70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34213 } } }
I20260929 02:17:24.708789 13657 sys_catalog.cc:458] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:24.710228 13660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:24.715870 13660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:24.724097 13660 catalog_manager.cc:1347] Generated new cluster ID: c949644873484b1aad880a380c9aefc8
I20260929 02:17:24.724200 13660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:24.739014 13660 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:24.739481 13660 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:24.756740 13660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f37aca36e15a4142bdc5b9d73690ae70: Generated new TSK 0
I20260929 02:17:24.757701 13660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:24.773504 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 13578
W20260929 02:17:24.774742 13640 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53279
W20260929 02:17:24.775462 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 13578: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:17:24.775604 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 13578
2026-09-29T02:17:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:24 dist-test-slave-9k0h krb5kdc[13676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:24 dist-test-slave-9k0h krb5kdc[13676](info): set up 2 sockets
Sep 29 02:17:24 dist-test-slave-9k0h krb5kdc[13676](info): commencing operation
krb5kdc: starting...
W20260929 02:17:25.882594 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:26 dist-test-slave-9k0h krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:17:26.146968 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:26.468761 13689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:26.469192 13689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:26.469372 13689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:26.476389 13689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:26.476536 13689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:26.476620 13689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:26.476706 13689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:26.476773 13689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:26.486241 13689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:26.487535 13689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:26.488813 13689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:26.500638 13694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:26.500672 13695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:26.500835 13697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:26.501642 13689 server_base.cc:1034] running on GCE node
I20260929 02:17:26.502439 13689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:26.503760 13689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:26.504957 13689 hybrid_clock.cc:648] HybridClock initialized: now 1790648246504913 us; error 34 us; skew 500 ppm
I20260929 02:17:26.505295 13689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:26 dist-test-slave-9k0h krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:26.514060 13689 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:17:26.515475 13689 webserver.cc:460] Webserver started at http://127.0.0.1:43767/ using document root <none> and password file <none>
I20260929 02:17:26.516100 13689 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:26.516196 13689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:26.516436 13689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:26.519357 13689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "c609eb4bc1384bf7a5f2d8aa2efb5174"
format_stamp: "Formatted at 2026-09-29 02:17:26 on dist-test-slave-9k0h"
server_key: "b0efba1252342c8645917a3bdc8c92d8"
server_key_iv: "5bebd29756a8f25292add7463193acb4"
server_key_version: "encryptionkey@0"
I20260929 02:17:26.520071 13689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "c609eb4bc1384bf7a5f2d8aa2efb5174"
format_stamp: "Formatted at 2026-09-29 02:17:26 on dist-test-slave-9k0h"
server_key: "b0efba1252342c8645917a3bdc8c92d8"
server_key_iv: "5bebd29756a8f25292add7463193acb4"
server_key_version: "encryptionkey@0"
I20260929 02:17:26.525005 13689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:17:26.528294 13704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:26.529549 13689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:17:26.529691 13689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "c609eb4bc1384bf7a5f2d8aa2efb5174"
format_stamp: "Formatted at 2026-09-29 02:17:26 on dist-test-slave-9k0h"
server_key: "b0efba1252342c8645917a3bdc8c92d8"
server_key_iv: "5bebd29756a8f25292add7463193acb4"
server_key_version: "encryptionkey@0"
I20260929 02:17:26.529844 13689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:26.547246 13689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:26.551232 13689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:26.551537 13689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:26.569869 13689 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41459
I20260929 02:17:26.569902 13755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41459 every 8 connection(s)
I20260929 02:17:26.571264 13689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:26.572086 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 13689
I20260929 02:17:26.572271 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:26.572814 25442 external_mini_cluster.cc:1428] Setting key 9ac59038781e06ac6fbb5011f6a6b8f2
I20260929 02:17:26.575721 13756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:17:26 dist-test-slave-9k0h krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:17:26.588603 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174: Bootstrap starting.
I20260929 02:17:26.591858 13756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:26.592873 13756 log.cc:824] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:26.595184 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:26.574600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36037 (local address 127.0.0.1:41459)
0929 02:17:26.575348 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:26.575359 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:26.575398 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:17:26.577009 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:26.577031 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:26.577402 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:26.577703 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:26.579556 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:26.580864 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:26.582652 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:26.583272 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:26.583403 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:26.588056 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:26.588064 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:26.588076 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:26.588109 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:26.591081 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:26.591885 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:26.591891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:26.591897 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:26.592096 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:26.592508 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:26.592513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:26.592517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:26.593030 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:26.593336 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:26.593814 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 02:17:26.594272 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":347,"threads_started":1}
I20260929 02:17:26.595701 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174: No bootstrap required, opened a new log
I20260929 02:17:26.598675 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:26.603538 13756 raft_consensus.cc:348] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c609eb4bc1384bf7a5f2d8aa2efb5174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41459 } }
I20260929 02:17:26.603894 13756 raft_consensus.cc:374] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:26.603994 13756 raft_consensus.cc:729] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c609eb4bc1384bf7a5f2d8aa2efb5174, State: Initialized, Role: FOLLOWER
I20260929 02:17:26.604501 13756 consensus_queue.cc:260] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c609eb4bc1384bf7a5f2d8aa2efb5174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41459 } }
I20260929 02:17:26.604736 13756 raft_consensus.cc:388] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:26.604874 13756 raft_consensus.cc:482] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:26.605042 13756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:26.607249 13756 raft_consensus.cc:504] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c609eb4bc1384bf7a5f2d8aa2efb5174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41459 } }
I20260929 02:17:26.607749 13756 leader_election.cc:302] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c609eb4bc1384bf7a5f2d8aa2efb5174; no voters:
I20260929 02:17:26.608381 13756 leader_election.cc:288] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:26.608443 13766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:26.610450 13766 raft_consensus.cc:686] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [term 1 LEADER]: Becoming Leader. State: Replica: c609eb4bc1384bf7a5f2d8aa2efb5174, State: Running, Role: LEADER
I20260929 02:17:26.611056 13766 consensus_queue.cc:237] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c609eb4bc1384bf7a5f2d8aa2efb5174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41459 } }
I20260929 02:17:26.613025 13756 sys_catalog.cc:564] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:26.615236 13768 sys_catalog.cc:455] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c609eb4bc1384bf7a5f2d8aa2efb5174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c609eb4bc1384bf7a5f2d8aa2efb5174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41459 } } }
I20260929 02:17:26.615669 13768 sys_catalog.cc:458] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:26.620236 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:26.602831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48377 (local address 127.0.0.1:41459)
0929 02:17:26.602971 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:26.602975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:26.603372 (+ 397us) server_negotiation.cc:407] Connection header received
0929 02:17:26.603553 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:26.603558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:26.603624 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:26.603729 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:26.605345 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:26.606111 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:26.607338 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:26.607870 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:26.608006 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:26.611559 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:26.611565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:26.611570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:26.611602 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:26.615640 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:26.616454 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:26.616459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:26.616462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:26.616696 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:26.617229 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:26.617237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:26.617242 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:26.617698 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:26.618040 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:26.619441 (+ 1401us) server_negotiation.cc:299] Negotiation successful
0929 02:17:26.619722 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:17:26.620894 13767 sys_catalog.cc:455] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c609eb4bc1384bf7a5f2d8aa2efb5174. Latest consensus state: current_term: 1 leader_uuid: "c609eb4bc1384bf7a5f2d8aa2efb5174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c609eb4bc1384bf7a5f2d8aa2efb5174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41459 } } }
I20260929 02:17:26.621284 13767 sys_catalog.cc:458] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:26.627126 13771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:26.632795 13771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:26.641358 13771 catalog_manager.cc:1347] Generated new cluster ID: d812a48c8f824bafb31204a7b49a8367
I20260929 02:17:26.641460 13771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:26.655725 13771 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:26.656951 13771 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:26.675951 13771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c609eb4bc1384bf7a5f2d8aa2efb5174: Generated new TSK 0
I20260929 02:17:26.676779 13771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:26.685029 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 13689
I20260929 02:17:26.686056 13751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:26.776517 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 13689
2026-09-29T02:17:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2001 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:26 dist-test-slave-9k0h krb5kdc[13788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:26 dist-test-slave-9k0h krb5kdc[13788](info): set up 2 sockets
Sep 29 02:17:26 dist-test-slave-9k0h krb5kdc[13788](info): commencing operation
krb5kdc: starting...
W20260929 02:17:27.890638 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:28 dist-test-slave-9k0h krb5kdc[13788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:17:28.155382 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:28.477285 13801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:28.477599 13801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:28.477775 13801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:28.484903 13801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:28.485050 13801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:28.485157 13801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:28.485271 13801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:28.485358 13801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:28.494827 13801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:28.496059 13801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:28.497351 13801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:28.508946 13809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:28.509051 13807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:28.509065 13806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:28.509895 13801 server_base.cc:1034] running on GCE node
I20260929 02:17:28.510779 13801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:28.512161 13801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:28.513365 13801 hybrid_clock.cc:648] HybridClock initialized: now 1790648248513331 us; error 27 us; skew 500 ppm
I20260929 02:17:28.513702 13801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:28 dist-test-slave-9k0h krb5kdc[13788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:28.517433 13801 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:17:28.519088 13801 webserver.cc:460] Webserver started at http://127.0.0.1:33363/ using document root <none> and password file <none>
I20260929 02:17:28.519701 13801 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:28.519842 13801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:28.520138 13801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:28.523149 13801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "3d0466dacdb94acbaa26f570901b337b"
format_stamp: "Formatted at 2026-09-29 02:17:28 on dist-test-slave-9k0h"
server_key: "07cdf26b13892b858f5a8c35a1e4ca67"
server_key_iv: "c6fc4a4f367c8b1c3b78033bfb849aaa"
server_key_version: "encryptionkey@0"
I20260929 02:17:28.523900 13801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "3d0466dacdb94acbaa26f570901b337b"
format_stamp: "Formatted at 2026-09-29 02:17:28 on dist-test-slave-9k0h"
server_key: "07cdf26b13892b858f5a8c35a1e4ca67"
server_key_iv: "c6fc4a4f367c8b1c3b78033bfb849aaa"
server_key_version: "encryptionkey@0"
I20260929 02:17:28.528919 13801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:17:28.532388 13816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:28.533674 13801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:17:28.533864 13801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "3d0466dacdb94acbaa26f570901b337b"
format_stamp: "Formatted at 2026-09-29 02:17:28 on dist-test-slave-9k0h"
server_key: "07cdf26b13892b858f5a8c35a1e4ca67"
server_key_iv: "c6fc4a4f367c8b1c3b78033bfb849aaa"
server_key_version: "encryptionkey@0"
I20260929 02:17:28.534024 13801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:28.558282 13801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:28.562325 13801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:28.562602 13801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:28.580775 13801 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35247
I20260929 02:17:28.580856 13867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35247 every 8 connection(s)
I20260929 02:17:28.582080 13801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:28.585619 13868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:28.591094 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 13801
I20260929 02:17:28.591280 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:28.591712 25442 external_mini_cluster.cc:1428] Setting key 2de7d84139a301afa570a61f8bcee04d
I20260929 02:17:28.596310 13868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b: Bootstrap starting.
I20260929 02:17:28.599344 13868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:28.600343 13868 log.cc:824] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:28.603103 13868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b: No bootstrap required, opened a new log
I20260929 02:17:28.604079 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:28.593421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44833 (local address 127.0.0.1:35247)
0929 02:17:28.594256 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:28.594270 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:28.594314 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:17:28.595853 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:28.595874 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:28.596234 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:28.596622 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:28.597113 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:28.598514 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:28.599858 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:28.600549 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:28.600724 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:28.601853 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:28.601860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:28.601872 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:28.602435 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:28.602504 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:28.602527 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:28.602649 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:28.602872 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 02:17:28.603120 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":345,"threads_started":1}
I20260929 02:17:28.607519 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:28.610843 13868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0466dacdb94acbaa26f570901b337b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35247 } }
I20260929 02:17:28.611178 13868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:28.611279 13868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0466dacdb94acbaa26f570901b337b, State: Initialized, Role: FOLLOWER
I20260929 02:17:28.612298 13868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0466dacdb94acbaa26f570901b337b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35247 } }
I20260929 02:17:28.612627 13868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:28.612787 13868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:28.612977 13868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:28.616055 13868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0466dacdb94acbaa26f570901b337b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35247 } }
I20260929 02:17:28.616627 13868 leader_election.cc:302] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d0466dacdb94acbaa26f570901b337b; no voters:
I20260929 02:17:28.617375 13868 leader_election.cc:288] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:28.617481 13878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:28.618676 13878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [term 1 LEADER]: Becoming Leader. State: Replica: 3d0466dacdb94acbaa26f570901b337b, State: Running, Role: LEADER
I20260929 02:17:28.618860 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:28.611618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42017 (local address 127.0.0.1:35247)
0929 02:17:28.611828 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:28.611836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:28.612254 (+ 418us) server_negotiation.cc:407] Connection header received
0929 02:17:28.612456 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:28.612460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:28.612517 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:28.612631 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:28.613240 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:28.613965 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:28.615812 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:28.616321 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:28.616410 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:28.617489 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:28.617495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:28.617502 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:28.617525 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:28.617569 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:28.617591 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:28.617706 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:28.618242 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 02:17:28.618361 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:17:28.619252 13878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0466dacdb94acbaa26f570901b337b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35247 } }
I20260929 02:17:28.619975 13868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:28.622056 13879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d0466dacdb94acbaa26f570901b337b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0466dacdb94acbaa26f570901b337b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35247 } } }
I20260929 02:17:28.622082 13880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0466dacdb94acbaa26f570901b337b. Latest consensus state: current_term: 1 leader_uuid: "3d0466dacdb94acbaa26f570901b337b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0466dacdb94acbaa26f570901b337b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35247 } } }
I20260929 02:17:28.622570 13879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:28.622570 13880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:28.624815 13883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:28.630098 13883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:28.639297 13883 catalog_manager.cc:1347] Generated new cluster ID: 6b134ba6ab874cb4b214d9d133065a74
I20260929 02:17:28.639423 13883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:28.651117 13883 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:28.652359 13883 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:28.664204 13883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d0466dacdb94acbaa26f570901b337b: Generated new TSK 0
I20260929 02:17:28.665220 13883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:28.679230 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 13801
W20260929 02:17:28.680286 13863 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44833
W20260929 02:17:28.681275 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 13801: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:17:28.681419 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 13801
2026-09-29T02:17:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1904 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:28 dist-test-slave-9k0h krb5kdc[13899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:28 dist-test-slave-9k0h krb5kdc[13899](info): set up 2 sockets
Sep 29 02:17:28 dist-test-slave-9k0h krb5kdc[13899](info): commencing operation
krb5kdc: starting...
W20260929 02:17:29.786828 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:29 dist-test-slave-9k0h krb5kdc[13899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:17:30.053297 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:30.383549 13912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:30.383833 13912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:30.383926 13912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:30.391069 13912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:30.391170 13912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:30.391222 13912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:30.391280 13912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:30.391323 13912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:30.400810 13912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:30.401996 13912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:30.403259 13912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:30.415871 13917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:30.415937 13920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:30.415956 13918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:30.416630 13912 server_base.cc:1034] running on GCE node
I20260929 02:17:30.417466 13912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:30.418921 13912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:30.420116 13912 hybrid_clock.cc:648] HybridClock initialized: now 1790648250420062 us; error 57 us; skew 500 ppm
I20260929 02:17:30.420452 13912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:30 dist-test-slave-9k0h krb5kdc[13899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:30.423503 13912 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:17:30.424893 13912 webserver.cc:460] Webserver started at http://127.0.0.1:39341/ using document root <none> and password file <none>
I20260929 02:17:30.425542 13912 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:30.425694 13912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:30.426004 13912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:30.429037 13912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "0950b8e378b24de599712483d816c372"
format_stamp: "Formatted at 2026-09-29 02:17:30 on dist-test-slave-9k0h"
server_key: "ebbbea1b074a5200cddddcb049ad9241"
server_key_iv: "dc79e61898bf11d60105c5982b015119"
server_key_version: "encryptionkey@0"
I20260929 02:17:30.429802 13912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "0950b8e378b24de599712483d816c372"
format_stamp: "Formatted at 2026-09-29 02:17:30 on dist-test-slave-9k0h"
server_key: "ebbbea1b074a5200cddddcb049ad9241"
server_key_iv: "dc79e61898bf11d60105c5982b015119"
server_key_version: "encryptionkey@0"
I20260929 02:17:30.434823 13912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:17:30.438143 13927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:30.439467 13912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:17:30.439642 13912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "0950b8e378b24de599712483d816c372"
format_stamp: "Formatted at 2026-09-29 02:17:30 on dist-test-slave-9k0h"
server_key: "ebbbea1b074a5200cddddcb049ad9241"
server_key_iv: "dc79e61898bf11d60105c5982b015119"
server_key_version: "encryptionkey@0"
I20260929 02:17:30.439870 13912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:30.445132 13912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:30.448840 13912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:30.449123 13912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:30.466137 13912 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43613
I20260929 02:17:30.466293 13978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43613 every 8 connection(s)
I20260929 02:17:30.467571 13912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:30.470003 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 13912
I20260929 02:17:30.470230 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:30.470793 25442 external_mini_cluster.cc:1428] Setting key c191c0312d60782ae7f7f69a6387b86b
I20260929 02:17:30.471649 13979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:30.479390 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:30.472405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34585 (local address 127.0.0.1:43613)
0929 02:17:30.473335 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:30.473350 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:30.473634 (+ 284us) server_negotiation.cc:407] Connection header received
0929 02:17:30.476203 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:30.476222 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:30.476522 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:30.476821 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:30.477829 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:30.477834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:30.477846 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:30.477859 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:30.477913 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:30.477945 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:30.478153 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:30.478340 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 02:17:30.478560 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":379,"threads_started":1}
I20260929 02:17:30.482933 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:30.484735 13979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372: Bootstrap starting.
I20260929 02:17:30.488307 13979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:30.489637 13979 log.cc:824] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:30.490020 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:30.487597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53207 (local address 127.0.0.1:43613)
0929 02:17:30.487800 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:30.487805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:30.488080 (+ 275us) server_negotiation.cc:407] Connection header received
0929 02:17:30.488254 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:30.488257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:30.488307 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:30.488384 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:30.489346 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:30.489350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:30.489353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:30.489366 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:30.489402 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:30.489416 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:30.489503 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:30.489648 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 02:17:30.489697 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 02:17:30.492529 13979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372: No bootstrap required, opened a new log
I20260929 02:17:30.499639 13979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0950b8e378b24de599712483d816c372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43613 } }
I20260929 02:17:30.499984 13979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:30.500056 13979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0950b8e378b24de599712483d816c372, State: Initialized, Role: FOLLOWER
I20260929 02:17:30.500583 13979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0950b8e378b24de599712483d816c372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43613 } }
I20260929 02:17:30.500772 13979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:30.500869 13979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:30.500984 13979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:30.503306 13979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0950b8e378b24de599712483d816c372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43613 } }
I20260929 02:17:30.503714 13979 leader_election.cc:302] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0950b8e378b24de599712483d816c372; no voters:
I20260929 02:17:30.504251 13979 leader_election.cc:288] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:30.504393 13989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:30.505123 13989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [term 1 LEADER]: Becoming Leader. State: Replica: 0950b8e378b24de599712483d816c372, State: Running, Role: LEADER
I20260929 02:17:30.505676 13989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0950b8e378b24de599712483d816c372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43613 } }
I20260929 02:17:30.506404 13979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:30.508493 13990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0950b8e378b24de599712483d816c372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0950b8e378b24de599712483d816c372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43613 } } }
I20260929 02:17:30.508893 13990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:30.508627 13991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0950b8e378b24de599712483d816c372. Latest consensus state: current_term: 1 leader_uuid: "0950b8e378b24de599712483d816c372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0950b8e378b24de599712483d816c372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43613 } } }
I20260929 02:17:30.509016 13991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:30.511804 13997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:30.518110 13997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:30.528474 13997 catalog_manager.cc:1347] Generated new cluster ID: 939afe1e68d949cb82c4d4dbf40d7ccb
I20260929 02:17:30.528591 13997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:30.538516 13997 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:30.538967 13997 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:30.546100 13997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0950b8e378b24de599712483d816c372: Generated new TSK 0
I20260929 02:17:30.547112 13997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:30.560009 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 13912
W20260929 02:17:30.561628 13974 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34585
W20260929 02:17:30.562407 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 13912: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:17:30.562609 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 13912
2026-09-29T02:17:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1880 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:30 dist-test-slave-9k0h krb5kdc[14010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:30 dist-test-slave-9k0h krb5kdc[14010](info): set up 2 sockets
Sep 29 02:17:30 dist-test-slave-9k0h krb5kdc[14010](info): commencing operation
krb5kdc: starting...
W20260929 02:17:31.678802 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:31 dist-test-slave-9k0h krb5kdc[14010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:17:31.947625 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:32.269563 14023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:32.269846 14023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:32.269936 14023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:32.276943 14023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:32.277056 14023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:32.277102 14023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:32.277176 14023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:32.277222 14023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:32.287338 14023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:32.288600 14023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:32.289855 14023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:32.301667 14028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:32.301720 14031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:32.301667 14029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:32.302685 14023 server_base.cc:1034] running on GCE node
I20260929 02:17:32.303501 14023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:32.304953 14023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:32.306329 14023 hybrid_clock.cc:648] HybridClock initialized: now 1790648252306286 us; error 59 us; skew 500 ppm
I20260929 02:17:32.306691 14023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:32 dist-test-slave-9k0h krb5kdc[14010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:32.317006 14023 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:17:32.318538 14023 webserver.cc:460] Webserver started at http://127.0.0.1:36951/ using document root <none> and password file <none>
I20260929 02:17:32.319217 14023 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:32.319345 14023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:32.319656 14023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:32.322724 14023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "47929d28dafd407faed883e1a1a1d1d1"
format_stamp: "Formatted at 2026-09-29 02:17:32 on dist-test-slave-9k0h"
server_key: "3d7586d2b3fcf71a74a3339fb2136a9b"
server_key_iv: "3ff0b2235425adc1e0264dec736337cb"
server_key_version: "encryptionkey@0"
I20260929 02:17:32.323479 14023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "47929d28dafd407faed883e1a1a1d1d1"
format_stamp: "Formatted at 2026-09-29 02:17:32 on dist-test-slave-9k0h"
server_key: "3d7586d2b3fcf71a74a3339fb2136a9b"
server_key_iv: "3ff0b2235425adc1e0264dec736337cb"
server_key_version: "encryptionkey@0"
I20260929 02:17:32.328397 14023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:17:32.331765 14038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:32.332978 14023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:17:32.333154 14023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "47929d28dafd407faed883e1a1a1d1d1"
format_stamp: "Formatted at 2026-09-29 02:17:32 on dist-test-slave-9k0h"
server_key: "3d7586d2b3fcf71a74a3339fb2136a9b"
server_key_iv: "3ff0b2235425adc1e0264dec736337cb"
server_key_version: "encryptionkey@0"
I20260929 02:17:32.333343 14023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:32.350865 14023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:32.355125 14023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:32.355417 14023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:32.373035 14023 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39245
I20260929 02:17:32.373075 14089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39245 every 8 connection(s)
I20260929 02:17:32.374394 14023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:32.377890 14090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:32.383064 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 14023
I20260929 02:17:32.383268 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:32.383731 25442 external_mini_cluster.cc:1428] Setting key 175facf899d6dd305e8919b5983940b1
I20260929 02:17:32.388499 14090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1: Bootstrap starting.
I20260929 02:17:32.391592 14090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:32.392597 14090 log.cc:824] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:32.395318 14090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1: No bootstrap required, opened a new log
Sep 29 02:17:32 dist-test-slave-9k0h krb5kdc[14010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:17:32.405186 14090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47929d28dafd407faed883e1a1a1d1d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39245 } }
I20260929 02:17:32.405603 14090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:32.405717 14090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47929d28dafd407faed883e1a1a1d1d1, State: Initialized, Role: FOLLOWER
I20260929 02:17:32.406271 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:32.385524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:34275 (local address 127.0.0.1:39245)
0929 02:17:32.386322 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:32.386333 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:32.386373 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:17:32.388004 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:32.388024 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:32.388372 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:32.388753 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:32.390093 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:32.391462 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:32.392842 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:32.393711 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:32.393925 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:32.397837 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:32.397848 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:32.397866 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:32.397913 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:32.401815 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:32.402602 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:32.402610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:32.402618 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:32.402851 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:32.403261 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:32.403268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:32.403276 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:32.403947 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:32.404275 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:32.404654 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 02:17:32.404977 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":349,"threads_started":1}
I20260929 02:17:32.406369 14090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47929d28dafd407faed883e1a1a1d1d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39245 } }
I20260929 02:17:32.406642 14090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:32.406818 14090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:32.407004 14090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:32.409466 14090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47929d28dafd407faed883e1a1a1d1d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39245 } }
I20260929 02:17:32.409937 14090 leader_election.cc:302] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47929d28dafd407faed883e1a1a1d1d1; no voters:
I20260929 02:17:32.410095 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:32.410560 14090 leader_election.cc:288] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:32.410684 14095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:32.412047 14095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [term 1 LEADER]: Becoming Leader. State: Replica: 47929d28dafd407faed883e1a1a1d1d1, State: Running, Role: LEADER
I20260929 02:17:32.412515 14090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:32.412561 14095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47929d28dafd407faed883e1a1a1d1d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39245 } }
I20260929 02:17:32.415308 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47929d28dafd407faed883e1a1a1d1d1. Latest consensus state: current_term: 1 leader_uuid: "47929d28dafd407faed883e1a1a1d1d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47929d28dafd407faed883e1a1a1d1d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39245 } } }
I20260929 02:17:32.415743 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:32.416260 14096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47929d28dafd407faed883e1a1a1d1d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47929d28dafd407faed883e1a1a1d1d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39245 } } }
I20260929 02:17:32.416523 14096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:32.418013 14105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:32.422986 14105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:32.449460 14105 catalog_manager.cc:1347] Generated new cluster ID: 7ba7b54908b2411c939e99b628c9d24e
I20260929 02:17:32.449599 14105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:32.452672 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:32.420325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:37711 (local address 127.0.0.1:39245)
0929 02:17:32.425054 (+ 4729us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:32.425062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:32.430532 (+ 5470us) server_negotiation.cc:407] Connection header received
0929 02:17:32.430618 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:32.430626 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:32.430732 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:32.430851 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:32.433172 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:32.434698 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:32.437223 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:32.438091 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:32.438307 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:32.442501 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:32.442512 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:32.442519 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:32.442569 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:32.448392 (+ 5823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:32.449453 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:32.449459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:32.449462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:32.449603 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:32.450212 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:32.450217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:32.450221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:32.450638 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:32.450887 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:32.452077 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0929 02:17:32.452295 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4583}
I20260929 02:17:32.467396 14105 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:32.468828 14105 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:32.480001 14105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47929d28dafd407faed883e1a1a1d1d1: Generated new TSK 0
I20260929 02:17:32.480806 14105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:32.493064 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14023
I20260929 02:17:32.494185 14085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:32.586637 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 14023
W20260929 02:17:32.603688 14008 connection.cc:535] client connection to 127.0.0.1:39245 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39245): Connection reset by peer (error 104)
2026-09-29T02:17:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:32 dist-test-slave-9k0h krb5kdc[14122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:32 dist-test-slave-9k0h krb5kdc[14122](info): set up 2 sockets
Sep 29 02:17:32 dist-test-slave-9k0h krb5kdc[14122](info): commencing operation
krb5kdc: starting...
W20260929 02:17:33.706712 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:33 dist-test-slave-9k0h krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:17:33.985570 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:34.325196 14135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:34.325531 14135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:34.325721 14135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:34.333096 14135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:34.333256 14135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:34.333371 14135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:34.333493 14135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:34.333578 14135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:34.343561 14135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:34.344861 14135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:34.346256 14135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:34.358356 14140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:34.358417 14141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:34.358421 14143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:34.359619 14135 server_base.cc:1034] running on GCE node
I20260929 02:17:34.360379 14135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:34.361744 14135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:34.362932 14135 hybrid_clock.cc:648] HybridClock initialized: now 1790648254362863 us; error 64 us; skew 500 ppm
I20260929 02:17:34.363271 14135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:34 dist-test-slave-9k0h krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:34.366514 14135 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:17:34.367856 14135 webserver.cc:460] Webserver started at http://127.0.0.1:36463/ using document root <none> and password file <none>
I20260929 02:17:34.368531 14135 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:34.368685 14135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:34.368980 14135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:34.371892 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "84c2351a94c74046aea7e9c0c792028c"
format_stamp: "Formatted at 2026-09-29 02:17:34 on dist-test-slave-9k0h"
server_key: "79bf4695e006fccf9d8e12af560e9e02"
server_key_iv: "cfe4252216b0e3ee533057ec2fa53e4d"
server_key_version: "encryptionkey@0"
I20260929 02:17:34.372658 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "84c2351a94c74046aea7e9c0c792028c"
format_stamp: "Formatted at 2026-09-29 02:17:34 on dist-test-slave-9k0h"
server_key: "79bf4695e006fccf9d8e12af560e9e02"
server_key_iv: "cfe4252216b0e3ee533057ec2fa53e4d"
server_key_version: "encryptionkey@0"
I20260929 02:17:34.377521 14135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 02:17:34.380816 14150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:34.382091 14135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:17:34.382294 14135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "84c2351a94c74046aea7e9c0c792028c"
format_stamp: "Formatted at 2026-09-29 02:17:34 on dist-test-slave-9k0h"
server_key: "79bf4695e006fccf9d8e12af560e9e02"
server_key_iv: "cfe4252216b0e3ee533057ec2fa53e4d"
server_key_version: "encryptionkey@0"
I20260929 02:17:34.382512 14135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:34.400040 14135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:34.404305 14135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:34.404585 14135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:34.422652 14135 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40057
I20260929 02:17:34.422732 14201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40057 every 8 connection(s)
I20260929 02:17:34.424003 14135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:34.427592 14202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:34.433032 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 14135
I20260929 02:17:34.433219 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:34.433743 25442 external_mini_cluster.cc:1428] Setting key 53956cbfca2cd6e5b7a438857c24b428
I20260929 02:17:34.438365 14202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c: Bootstrap starting.
I20260929 02:17:34.441260 14202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:34.442245 14202 log.cc:824] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:34.445255 14202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c: No bootstrap required, opened a new log
I20260929 02:17:34.446498 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:34.435554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:45947 (local address 127.0.0.1:40057)
0929 02:17:34.436348 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:34.436358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:34.436401 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:17:34.438015 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:34.438036 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:34.438518 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:34.438885 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:34.439382 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:34.440679 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:34.442051 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:34.442972 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:34.443169 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:34.444291 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:34.444298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:34.444311 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:34.444816 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:34.444889 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:34.444912 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:34.445030 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:34.445331 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 02:17:34.445504 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":349,"threads_started":1}
I20260929 02:17:34.450136 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:34.454345 14202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c2351a94c74046aea7e9c0c792028c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40057 } }
I20260929 02:17:34.454782 14202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:34.454883 14202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c2351a94c74046aea7e9c0c792028c, State: Initialized, Role: FOLLOWER
I20260929 02:17:34.455493 14202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c2351a94c74046aea7e9c0c792028c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40057 } }
I20260929 02:17:34.455754 14202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:34.455910 14202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:34.456084 14202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:34.459012 14202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c2351a94c74046aea7e9c0c792028c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40057 } }
I20260929 02:17:34.459498 14202 leader_election.cc:302] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84c2351a94c74046aea7e9c0c792028c; no voters:
I20260929 02:17:34.460224 14202 leader_election.cc:288] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:34.460345 14212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:34.461431 14212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [term 1 LEADER]: Becoming Leader. State: Replica: 84c2351a94c74046aea7e9c0c792028c, State: Running, Role: LEADER
I20260929 02:17:34.462139 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:34.455155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:55909 (local address 127.0.0.1:40057)
0929 02:17:34.455315 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:34.455322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:34.455916 (+ 594us) server_negotiation.cc:407] Connection header received
0929 02:17:34.456118 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:34.456123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:34.456216 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:34.456326 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:34.456736 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:34.457880 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:34.459216 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:34.459777 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:34.459907 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:34.461187 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:34.461193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:34.461196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:34.461214 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:34.461276 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:34.461301 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:34.461405 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:34.461761 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 02:17:34.461809 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:17:34.462065 14212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c2351a94c74046aea7e9c0c792028c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40057 } }
I20260929 02:17:34.462558 14202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:34.465214 14214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84c2351a94c74046aea7e9c0c792028c. Latest consensus state: current_term: 1 leader_uuid: "84c2351a94c74046aea7e9c0c792028c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c2351a94c74046aea7e9c0c792028c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40057 } } }
I20260929 02:17:34.465566 14214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:34.465304 14213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84c2351a94c74046aea7e9c0c792028c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c2351a94c74046aea7e9c0c792028c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40057 } } }
I20260929 02:17:34.465687 14213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:34.469102 14218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:34.475214 14218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:34.483693 14218 catalog_manager.cc:1347] Generated new cluster ID: 9bb468e3ada141b6b910a843717d16d6
I20260929 02:17:34.483793 14218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:34.511292 14218 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:34.512563 14218 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:34.519145 14218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84c2351a94c74046aea7e9c0c792028c: Generated new TSK 0
I20260929 02:17:34.519938 14218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:34.534251 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14135
W20260929 02:17:34.535655 14197 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.135:45947
W20260929 02:17:34.536393 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14135: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:17:34.536538 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 14135
W20260929 02:17:34.554313 14120 connection.cc:535] client connection to 127.0.0.1:40057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40057): Connection reset by peer (error 104)
2026-09-29T02:17:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:34 dist-test-slave-9k0h krb5kdc[14233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:34 dist-test-slave-9k0h krb5kdc[14233](info): set up 2 sockets
Sep 29 02:17:34 dist-test-slave-9k0h krb5kdc[14233](info): commencing operation
krb5kdc: starting...
W20260929 02:17:35.634711 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.006s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:35 dist-test-slave-9k0h krb5kdc[14233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:17:35.890899 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:36.210983 14246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:36.211305 14246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:36.211472 14246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:36.218621 14246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:36.218765 14246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:36.218866 14246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:36.218969 14246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:36.219041 14246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:36.228569 14246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:36.229844 14246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:36.231149 14246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:36.242578 14252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:36.242739 14251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:36.242627 14254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:36.243706 14246 server_base.cc:1034] running on GCE node
I20260929 02:17:36.244416 14246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:36.245694 14246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:36.246889 14246 hybrid_clock.cc:648] HybridClock initialized: now 1790648256246852 us; error 32 us; skew 500 ppm
I20260929 02:17:36.247223 14246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:36 dist-test-slave-9k0h krb5kdc[14233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:36.250250 14246 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:17:36.251658 14246 webserver.cc:460] Webserver started at http://127.0.0.1:41805/ using document root <none> and password file <none>
I20260929 02:17:36.252296 14246 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:36.252395 14246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:36.252713 14246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:36.255589 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "d17dc39e329741809df22861929f40da"
format_stamp: "Formatted at 2026-09-29 02:17:36 on dist-test-slave-9k0h"
server_key: "2c600ee14c48bc2d3fe76dff8d10ce2d"
server_key_iv: "1ce90ca66f6868ed4f1100a11b84f009"
server_key_version: "encryptionkey@0"
I20260929 02:17:36.256351 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "d17dc39e329741809df22861929f40da"
format_stamp: "Formatted at 2026-09-29 02:17:36 on dist-test-slave-9k0h"
server_key: "2c600ee14c48bc2d3fe76dff8d10ce2d"
server_key_iv: "1ce90ca66f6868ed4f1100a11b84f009"
server_key_version: "encryptionkey@0"
I20260929 02:17:36.261099 14246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:17:36.264274 14261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:36.265492 14246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 02:17:36.265655 14246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "d17dc39e329741809df22861929f40da"
format_stamp: "Formatted at 2026-09-29 02:17:36 on dist-test-slave-9k0h"
server_key: "2c600ee14c48bc2d3fe76dff8d10ce2d"
server_key_iv: "1ce90ca66f6868ed4f1100a11b84f009"
server_key_version: "encryptionkey@0"
I20260929 02:17:36.265843 14246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:36.271461 14246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:36.275202 14246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:36.275480 14246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:36.293499 14246 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45197
I20260929 02:17:36.293543 14312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45197 every 8 connection(s)
I20260929 02:17:36.295010 14246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:36.296970 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 14246
I20260929 02:17:36.297147 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:36.297720 25442 external_mini_cluster.cc:1428] Setting key 064a24cb6662960715cd47d5a73ae407
I20260929 02:17:36.299360 14313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:36.305137 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:36.299525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:60513 (local address 127.0.0.1:45197)
0929 02:17:36.300298 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:36.300308 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:36.300348 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:17:36.301845 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:36.301865 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:36.302259 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:36.302577 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:36.303629 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:36.303634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:36.303646 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:36.303660 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:36.303729 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:36.303747 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:36.303905 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:36.304030 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 02:17:36.304239 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":362,"threads_started":1}
I20260929 02:17:36.308631 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:36.311677 14313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da: Bootstrap starting.
I20260929 02:17:36.315367 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:36.312950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:58883 (local address 127.0.0.1:45197)
0929 02:17:36.313157 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:36.313163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:36.313470 (+ 307us) server_negotiation.cc:407] Connection header received
0929 02:17:36.313632 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:36.313636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:36.313685 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:36.313761 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:36.314726 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:36.314729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:36.314732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:36.314745 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:36.314779 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:36.314793 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:36.314875 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:36.315022 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 02:17:36.315064 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 02:17:36.315372 14313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:36.316519 14313 log.cc:824] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:36.319000 14313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da: No bootstrap required, opened a new log
I20260929 02:17:36.325851 14313 raft_consensus.cc:348] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17dc39e329741809df22861929f40da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45197 } }
I20260929 02:17:36.326258 14313 raft_consensus.cc:374] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:36.326393 14313 raft_consensus.cc:729] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17dc39e329741809df22861929f40da, State: Initialized, Role: FOLLOWER
I20260929 02:17:36.326937 14313 consensus_queue.cc:260] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17dc39e329741809df22861929f40da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45197 } }
I20260929 02:17:36.327183 14313 raft_consensus.cc:388] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:36.327320 14313 raft_consensus.cc:482] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:36.327499 14313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:36.329653 14313 raft_consensus.cc:504] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17dc39e329741809df22861929f40da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45197 } }
I20260929 02:17:36.330139 14313 leader_election.cc:302] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d17dc39e329741809df22861929f40da; no voters:
I20260929 02:17:36.330772 14313 leader_election.cc:288] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:36.330904 14323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:36.331547 14323 raft_consensus.cc:686] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [term 1 LEADER]: Becoming Leader. State: Replica: d17dc39e329741809df22861929f40da, State: Running, Role: LEADER
I20260929 02:17:36.332082 14323 consensus_queue.cc:237] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17dc39e329741809df22861929f40da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45197 } }
I20260929 02:17:36.332973 14313 sys_catalog.cc:564] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:36.335016 14324 sys_catalog.cc:455] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d17dc39e329741809df22861929f40da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17dc39e329741809df22861929f40da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45197 } } }
I20260929 02:17:36.334919 14325 sys_catalog.cc:455] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [sys.catalog]: SysCatalogTable state changed. Reason: New leader d17dc39e329741809df22861929f40da. Latest consensus state: current_term: 1 leader_uuid: "d17dc39e329741809df22861929f40da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17dc39e329741809df22861929f40da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45197 } } }
I20260929 02:17:36.335449 14324 sys_catalog.cc:458] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:36.335464 14325 sys_catalog.cc:458] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:36.337652 14330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:36.343432 14330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:36.351884 14330 catalog_manager.cc:1347] Generated new cluster ID: a8e52b5a1e6d43f0b2ac99caaa259ddc
I20260929 02:17:36.351998 14330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:36.365468 14330 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:36.365970 14330 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:36.376909 14330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d17dc39e329741809df22861929f40da: Generated new TSK 0
I20260929 02:17:36.377725 14330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:36.381476 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14246
W20260929 02:17:36.382488 14308 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.135:60513
W20260929 02:17:36.383220 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14246: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:17:36.383376 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 14246
2026-09-29T02:17:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1842 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:36 dist-test-slave-9k0h krb5kdc[14344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:36 dist-test-slave-9k0h krb5kdc[14344](info): set up 2 sockets
Sep 29 02:17:36 dist-test-slave-9k0h krb5kdc[14344](info): commencing operation
krb5kdc: starting...
W20260929 02:17:37.486676 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.007s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:37 dist-test-slave-9k0h krb5kdc[14344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:17:37.757045 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:38.085018 14357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:38.085305 14357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:38.085497 14357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:38.092598 14357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:38.092729 14357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:38.092832 14357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:38.092938 14357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:38.093044 14357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:38.102892 14357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43169
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:38.104153 14357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:38.105443 14357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:38.116770 14362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:38.116729 14363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:38.116814 14365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:38.117918 14357 server_base.cc:1034] running on GCE node
I20260929 02:17:38.118701 14357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:38.119964 14357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:38.121189 14357 hybrid_clock.cc:648] HybridClock initialized: now 1790648258121151 us; error 48 us; skew 500 ppm
I20260929 02:17:38.121528 14357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:38 dist-test-slave-9k0h krb5kdc[14344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:38.138051 14357 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:17:38.139539 14357 webserver.cc:460] Webserver started at http://127.0.0.1:41253/ using document root <none> and password file <none>
I20260929 02:17:38.140232 14357 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:38.140375 14357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:38.140676 14357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:38.143667 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "f7788900e68544cb918e61fc26b4a038"
format_stamp: "Formatted at 2026-09-29 02:17:38 on dist-test-slave-9k0h"
server_key: "e2d7b3a4e5dff88288d055c81645a99a"
server_key_iv: "8cc8701dba421b4d4557687339c8c7e9"
server_key_version: "encryptionkey@0"
I20260929 02:17:38.144443 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "f7788900e68544cb918e61fc26b4a038"
format_stamp: "Formatted at 2026-09-29 02:17:38 on dist-test-slave-9k0h"
server_key: "e2d7b3a4e5dff88288d055c81645a99a"
server_key_iv: "8cc8701dba421b4d4557687339c8c7e9"
server_key_version: "encryptionkey@0"
I20260929 02:17:38.149806 14357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:17:38.153213 14372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:38.154598 14357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:17:38.154752 14357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "f7788900e68544cb918e61fc26b4a038"
format_stamp: "Formatted at 2026-09-29 02:17:38 on dist-test-slave-9k0h"
server_key: "e2d7b3a4e5dff88288d055c81645a99a"
server_key_iv: "8cc8701dba421b4d4557687339c8c7e9"
server_key_version: "encryptionkey@0"
I20260929 02:17:38.154907 14357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:38.179641 14357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:38.183444 14357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:38.183677 14357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:38.201669 14357 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43169
I20260929 02:17:38.201712 14423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43169 every 8 connection(s)
I20260929 02:17:38.202994 14357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:38.206523 14424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:38.213281 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 14357
I20260929 02:17:38.213493 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:38.213948 25442 external_mini_cluster.cc:1428] Setting key c8fd998ecff5d2a8a2fa7fe23c6f83b0
I20260929 02:17:38.216555 14424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038: Bootstrap starting.
I20260929 02:17:38.219866 14424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:38.221139 14424 log.cc:824] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:38.224112 14424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038: No bootstrap required, opened a new log
Sep 29 02:17:38 dist-test-slave-9k0h krb5kdc[14344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:17:38.233392 14424 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7788900e68544cb918e61fc26b4a038" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43169 } }
I20260929 02:17:38.233894 14424 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:38.234021 14424 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7788900e68544cb918e61fc26b4a038, State: Initialized, Role: FOLLOWER
I20260929 02:17:38.234586 14424 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7788900e68544cb918e61fc26b4a038" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43169 } }
I20260929 02:17:38.234855 14424 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:38.234997 14424 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:38.235195 14424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:38.237152 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:38.215704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:49883 (local address 127.0.0.1:43169)
0929 02:17:38.216481 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:38.216492 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:38.216532 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:17:38.218289 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:38.218310 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:38.218671 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:38.218978 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:38.220771 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:38.222226 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:38.223481 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:38.224474 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:38.224691 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:38.228531 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:38.228545 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:38.228564 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:38.228609 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:38.232573 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:38.233328 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:38.233337 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:38.233345 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:38.233568 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:38.234078 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:38.234082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:38.234084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:38.234893 (+ 809us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:38.235184 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:38.235746 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 02:17:38.236072 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":327,"threads_started":1}
I20260929 02:17:38.237824 14424 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7788900e68544cb918e61fc26b4a038" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43169 } }
I20260929 02:17:38.238361 14424 leader_election.cc:302] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7788900e68544cb918e61fc26b4a038; no voters:
I20260929 02:17:38.239029 14424 leader_election.cc:288] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:38.239156 14429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:38.240521 14429 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [term 1 LEADER]: Becoming Leader. State: Replica: f7788900e68544cb918e61fc26b4a038, State: Running, Role: LEADER
I20260929 02:17:38.241093 14424 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:38.241173 14429 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7788900e68544cb918e61fc26b4a038" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43169 } }
I20260929 02:17:38.241583 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:38.247373 14430 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7788900e68544cb918e61fc26b4a038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7788900e68544cb918e61fc26b4a038" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43169 } } }
I20260929 02:17:38.247705 14430 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:38.247323 14431 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7788900e68544cb918e61fc26b4a038. Latest consensus state: current_term: 1 leader_uuid: "f7788900e68544cb918e61fc26b4a038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7788900e68544cb918e61fc26b4a038" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43169 } } }
I20260929 02:17:38.248278 14431 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:38.258831 14438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:38.265108 14438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:38.279570 14438 catalog_manager.cc:1347] Generated new cluster ID: 1ac88a4dae154a25b8f3088f9a20055a
I20260929 02:17:38.279743 14438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:38.288218 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:38.260289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:39507 (local address 127.0.0.1:43169)
0929 02:17:38.260508 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:38.260516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:38.270496 (+ 9980us) server_negotiation.cc:407] Connection header received
0929 02:17:38.270713 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:38.270719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:38.270810 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:38.270923 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:38.272828 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:38.273936 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:38.276360 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:38.277269 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:38.277393 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:38.280492 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:38.280502 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:38.280512 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:38.280556 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:38.284453 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:38.285433 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:38.285439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:38.285443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:38.285598 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:38.286263 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:38.286268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:38.286272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:38.286686 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:38.286928 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:38.287636 (+ 708us) server_negotiation.cc:299] Negotiation successful
0929 02:17:38.287828 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:17:38.294346 14438 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:38.295622 14438 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:38.303220 14438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7788900e68544cb918e61fc26b4a038: Generated new TSK 0
I20260929 02:17:38.304165 14438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:38.314299 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14357
I20260929 02:17:38.315531 14419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:38.402768 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 14357
W20260929 02:17:38.419819 14342 connection.cc:535] client connection to 127.0.0.1:43169 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43169): Connection reset by peer (error 104)
2026-09-29T02:17:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2022 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:38 dist-test-slave-9k0h krb5kdc[14456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:38 dist-test-slave-9k0h krb5kdc[14456](info): set up 2 sockets
Sep 29 02:17:38 dist-test-slave-9k0h krb5kdc[14456](info): commencing operation
krb5kdc: starting...
W20260929 02:17:39.510725 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:39 dist-test-slave-9k0h krb5kdc[14456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:17:39.793552 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:40.112959 14469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:40.113291 14469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:40.113451 14469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:40.120575 14469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:40.120730 14469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:40.120827 14469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:40.120939 14469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:40.121018 14469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:40.130558 14469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:40.131807 14469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:40.133052 14469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:40.144536 14474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:40.144512 14477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:40.144459 14475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:40.145570 14469 server_base.cc:1034] running on GCE node
I20260929 02:17:40.146341 14469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:40.147732 14469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:40.148933 14469 hybrid_clock.cc:648] HybridClock initialized: now 1790648260148889 us; error 33 us; skew 500 ppm
I20260929 02:17:40.149266 14469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:40 dist-test-slave-9k0h krb5kdc[14456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:40.152213 14469 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:17:40.153597 14469 webserver.cc:460] Webserver started at http://127.0.0.1:41895/ using document root <none> and password file <none>
I20260929 02:17:40.154312 14469 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:40.154440 14469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:40.154742 14469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:40.157605 14469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "cdb7aaf8b3f74c7f8ef90761709d651e"
format_stamp: "Formatted at 2026-09-29 02:17:40 on dist-test-slave-9k0h"
server_key: "323dd08876010e37f8b28d73c11fcfcc"
server_key_iv: "ebd62d0736767dd46a4aec84000b852d"
server_key_version: "encryptionkey@0"
I20260929 02:17:40.158418 14469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "cdb7aaf8b3f74c7f8ef90761709d651e"
format_stamp: "Formatted at 2026-09-29 02:17:40 on dist-test-slave-9k0h"
server_key: "323dd08876010e37f8b28d73c11fcfcc"
server_key_iv: "ebd62d0736767dd46a4aec84000b852d"
server_key_version: "encryptionkey@0"
I20260929 02:17:40.163295 14469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:17:40.166667 14484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:40.168001 14469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:17:40.168218 14469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "cdb7aaf8b3f74c7f8ef90761709d651e"
format_stamp: "Formatted at 2026-09-29 02:17:40 on dist-test-slave-9k0h"
server_key: "323dd08876010e37f8b28d73c11fcfcc"
server_key_iv: "ebd62d0736767dd46a4aec84000b852d"
server_key_version: "encryptionkey@0"
I20260929 02:17:40.168429 14469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:40.183303 14469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:40.187371 14469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:40.187664 14469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:40.206033 14469 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38127
I20260929 02:17:40.206068 14535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38127 every 8 connection(s)
I20260929 02:17:40.207383 14469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:40.207800 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 14469
I20260929 02:17:40.207960 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:40.208555 25442 external_mini_cluster.cc:1428] Setting key 1817faa25c2b241dd298a759eb35e5e6
I20260929 02:17:40.211758 14536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:40.221925 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:40.210507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:47169 (local address 127.0.0.1:38127)
0929 02:17:40.211252 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:40.211264 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:40.211305 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:17:40.212947 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:40.212967 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:40.213332 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:40.213641 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:40.214283 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:40.215661 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:40.217521 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:40.218247 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:40.218376 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:40.219614 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:40.219622 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:40.219634 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:40.220177 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:40.220260 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:40.220287 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:40.220416 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:40.220739 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 02:17:40.220920 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":330,"threads_started":1}
I20260929 02:17:40.223690 14536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e: Bootstrap starting.
I20260929 02:17:40.225666 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:40.227200 14536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:40.228415 14536 log.cc:824] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:40.231688 14536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e: No bootstrap required, opened a new log
I20260929 02:17:40.237718 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:40.230930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:35993 (local address 127.0.0.1:38127)
0929 02:17:40.231112 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:40.231117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:40.231807 (+ 690us) server_negotiation.cc:407] Connection header received
0929 02:17:40.231987 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:40.231991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:40.232060 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:40.232206 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:40.232625 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:40.233770 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:40.235069 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:40.235777 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:40.235898 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:40.236923 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:40.236929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:40.236935 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:40.236954 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:40.236995 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:40.237015 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:40.237131 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:40.237296 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 02:17:40.237351 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:17:40.240566 14536 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb7aaf8b3f74c7f8ef90761709d651e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38127 } }
I20260929 02:17:40.240945 14536 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:40.241066 14536 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb7aaf8b3f74c7f8ef90761709d651e, State: Initialized, Role: FOLLOWER
I20260929 02:17:40.241626 14536 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb7aaf8b3f74c7f8ef90761709d651e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38127 } }
I20260929 02:17:40.241832 14536 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:40.241925 14536 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:40.242048 14536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:40.244604 14536 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb7aaf8b3f74c7f8ef90761709d651e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38127 } }
I20260929 02:17:40.245043 14536 leader_election.cc:302] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdb7aaf8b3f74c7f8ef90761709d651e; no voters:
I20260929 02:17:40.245606 14536 leader_election.cc:288] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:40.245709 14546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:40.247045 14546 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [term 1 LEADER]: Becoming Leader. State: Replica: cdb7aaf8b3f74c7f8ef90761709d651e, State: Running, Role: LEADER
I20260929 02:17:40.247529 14546 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb7aaf8b3f74c7f8ef90761709d651e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38127 } }
I20260929 02:17:40.247813 14536 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:40.250345 14548 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdb7aaf8b3f74c7f8ef90761709d651e. Latest consensus state: current_term: 1 leader_uuid: "cdb7aaf8b3f74c7f8ef90761709d651e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb7aaf8b3f74c7f8ef90761709d651e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38127 } } }
I20260929 02:17:40.250768 14548 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:40.250490 14547 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdb7aaf8b3f74c7f8ef90761709d651e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb7aaf8b3f74c7f8ef90761709d651e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38127 } } }
I20260929 02:17:40.250897 14547 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:40.251648 14554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:40.258074 14554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:40.266243 14554 catalog_manager.cc:1347] Generated new cluster ID: 23615afe98644cad9237fff6dbd6a4a4
I20260929 02:17:40.266371 14554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:40.289824 14554 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:40.291816 14554 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:40.312175 14554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdb7aaf8b3f74c7f8ef90761709d651e: Generated new TSK 0
I20260929 02:17:40.313002 14554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:40.331161 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14469
W20260929 02:17:40.332432 14531 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.135:47169
W20260929 02:17:40.333158 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14469: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:17:40.333321 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 14469
W20260929 02:17:40.347467 14454 connection.cc:535] client connection to 127.0.0.1:38127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38127): Connection reset by peer (error 104)
2026-09-29T02:17:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1927 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:40 dist-test-slave-9k0h krb5kdc[14567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:40 dist-test-slave-9k0h krb5kdc[14567](info): set up 2 sockets
Sep 29 02:17:40 dist-test-slave-9k0h krb5kdc[14567](info): commencing operation
krb5kdc: starting...
W20260929 02:17:41.430588 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.010s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:41 dist-test-slave-9k0h krb5kdc[14567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:17:41.699352 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:42.021595 14580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:42.021858 14580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:42.021947 14580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:42.029088 14580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:42.029212 14580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:42.029258 14580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:42.029318 14580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:42.029361 14580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:42.038859 14580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43607
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:42.040035 14580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:42.041215 14580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:42.052520 14585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:42.052690 14586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:42.053117 14588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:42.053718 14580 server_base.cc:1034] running on GCE node
I20260929 02:17:42.054497 14580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:42.055830 14580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:42.057021 14580 hybrid_clock.cc:648] HybridClock initialized: now 1790648262056969 us; error 52 us; skew 500 ppm
I20260929 02:17:42.057307 14580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:42 dist-test-slave-9k0h krb5kdc[14567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:42.060357 14580 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:17:42.061789 14580 webserver.cc:460] Webserver started at http://127.0.0.1:34163/ using document root <none> and password file <none>
I20260929 02:17:42.062527 14580 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:42.062633 14580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:42.062927 14580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:42.065743 14580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "018d32e43a4a4d47b33a6b6b9c5a8bae"
format_stamp: "Formatted at 2026-09-29 02:17:42 on dist-test-slave-9k0h"
server_key: "8eda81d59f0ba20e68978b85f4f3ee18"
server_key_iv: "6fddf78b24b82d68ab9eec37fe1528e5"
server_key_version: "encryptionkey@0"
I20260929 02:17:42.066514 14580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "018d32e43a4a4d47b33a6b6b9c5a8bae"
format_stamp: "Formatted at 2026-09-29 02:17:42 on dist-test-slave-9k0h"
server_key: "8eda81d59f0ba20e68978b85f4f3ee18"
server_key_iv: "6fddf78b24b82d68ab9eec37fe1528e5"
server_key_version: "encryptionkey@0"
I20260929 02:17:42.071532 14580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:17:42.074971 14595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:42.076349 14580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:17:42.076506 14580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "018d32e43a4a4d47b33a6b6b9c5a8bae"
format_stamp: "Formatted at 2026-09-29 02:17:42 on dist-test-slave-9k0h"
server_key: "8eda81d59f0ba20e68978b85f4f3ee18"
server_key_iv: "6fddf78b24b82d68ab9eec37fe1528e5"
server_key_version: "encryptionkey@0"
I20260929 02:17:42.076700 14580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:42.082559 14580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:42.087074 14580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:42.087334 14580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:42.106195 14580 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43607
I20260929 02:17:42.106200 14646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43607 every 8 connection(s)
I20260929 02:17:42.107721 14580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:42.111335 14647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:42.115392 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 14580
I20260929 02:17:42.115581 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:42.116012 25442 external_mini_cluster.cc:1428] Setting key a4f0abffb521882442bda1afded9c432
I20260929 02:17:42.121999 14647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae: Bootstrap starting.
I20260929 02:17:42.124308 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:42.117774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:44447 (local address 127.0.0.1:43607)
0929 02:17:42.118564 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:42.118575 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:42.118615 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:17:42.120944 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:42.120968 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:42.121330 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:42.121721 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:42.122798 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:42.122804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:42.122817 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:42.122830 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:42.122881 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:42.122899 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:42.123127 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:42.123252 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 02:17:42.123435 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":364,"threads_started":1}
I20260929 02:17:42.125315 14647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:42.126381 14647 log.cc:824] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:42.127931 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:42.129518 14647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae: No bootstrap required, opened a new log
I20260929 02:17:42.135488 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:42.133017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:56091 (local address 127.0.0.1:43607)
0929 02:17:42.133222 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:42.133229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:42.133534 (+ 305us) server_negotiation.cc:407] Connection header received
0929 02:17:42.133710 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:42.133714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:42.133759 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:42.133840 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:42.134826 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:42.134830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:42.134833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:42.134847 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:42.134880 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:42.134897 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:42.134988 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:42.135120 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 02:17:42.135166 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 02:17:42.139019 14647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018d32e43a4a4d47b33a6b6b9c5a8bae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260929 02:17:42.139477 14647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:42.139595 14647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 018d32e43a4a4d47b33a6b6b9c5a8bae, State: Initialized, Role: FOLLOWER
I20260929 02:17:42.140277 14647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018d32e43a4a4d47b33a6b6b9c5a8bae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260929 02:17:42.140511 14647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:42.140676 14647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:42.140849 14647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:42.143054 14647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018d32e43a4a4d47b33a6b6b9c5a8bae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260929 02:17:42.143544 14647 leader_election.cc:302] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 018d32e43a4a4d47b33a6b6b9c5a8bae; no voters:
I20260929 02:17:42.144112 14647 leader_election.cc:288] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:42.144213 14657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:42.145735 14657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [term 1 LEADER]: Becoming Leader. State: Replica: 018d32e43a4a4d47b33a6b6b9c5a8bae, State: Running, Role: LEADER
I20260929 02:17:42.146303 14647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:42.146267 14657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018d32e43a4a4d47b33a6b6b9c5a8bae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260929 02:17:42.149592 14659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 018d32e43a4a4d47b33a6b6b9c5a8bae. Latest consensus state: current_term: 1 leader_uuid: "018d32e43a4a4d47b33a6b6b9c5a8bae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018d32e43a4a4d47b33a6b6b9c5a8bae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } } }
I20260929 02:17:42.150060 14659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:42.149623 14658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "018d32e43a4a4d47b33a6b6b9c5a8bae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "018d32e43a4a4d47b33a6b6b9c5a8bae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } } }
I20260929 02:17:42.150804 14658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:42.152577 14666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:42.157891 14666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:42.166049 14666 catalog_manager.cc:1347] Generated new cluster ID: 11f0b52c24ce46d19df4bc7955a28aab
I20260929 02:17:42.166178 14666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:42.179937 14666 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:42.180430 14666 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:42.202101 14666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 018d32e43a4a4d47b33a6b6b9c5a8bae: Generated new TSK 0
I20260929 02:17:42.203192 14666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:42.209481 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14580
W20260929 02:17:42.210947 14642 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.135:44447
W20260929 02:17:42.211699 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14580: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:17:42.211825 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 14580
2026-09-29T02:17:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1881 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:42 dist-test-slave-9k0h krb5kdc[14678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:42 dist-test-slave-9k0h krb5kdc[14678](info): set up 2 sockets
Sep 29 02:17:42 dist-test-slave-9k0h krb5kdc[14678](info): commencing operation
krb5kdc: starting...
W20260929 02:17:43.346616 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.007s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:43 dist-test-slave-9k0h krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:17:43.607551 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:40749
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:40749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:43.933732 14691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:43.933995 14691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:43.934140 14691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:43.941092 14691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:43.941193 14691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:43.941308 14691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:43.941418 14691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:43.941506 14691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:43.950819 14691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:40749
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:40749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:43.952030 14691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:43.953277 14691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:43.964457 14696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:43.964536 14697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:43.964560 14699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:43.965425 14691 server_base.cc:1034] running on GCE node
I20260929 02:17:43.966264 14691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:43.967546 14691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:43.968777 14691 hybrid_clock.cc:648] HybridClock initialized: now 1790648263968745 us; error 45 us; skew 500 ppm
I20260929 02:17:43.969110 14691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:43 dist-test-slave-9k0h krb5kdc[14678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648263, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:43.975656 14691 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:17:43.977404 14691 webserver.cc:460] Webserver started at http://127.24.216.190:38265/ using document root <none> and password file <none>
I20260929 02:17:43.978087 14691 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:43.978281 14691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:43.978581 14691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:43.981485 14691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "f980356cc0f44aac9bf489701d71af0e"
format_stamp: "Formatted at 2026-09-29 02:17:43 on dist-test-slave-9k0h"
server_key: "4fbc210b3b2348d3ac980563f927e3c7"
server_key_iv: "2cb16efae74070ffa39137e2149f23dd"
server_key_version: "encryptionkey@0"
I20260929 02:17:43.982318 14691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "f980356cc0f44aac9bf489701d71af0e"
format_stamp: "Formatted at 2026-09-29 02:17:43 on dist-test-slave-9k0h"
server_key: "4fbc210b3b2348d3ac980563f927e3c7"
server_key_iv: "2cb16efae74070ffa39137e2149f23dd"
server_key_version: "encryptionkey@0"
I20260929 02:17:43.987427 14691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:17:43.990792 14706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:43.991972 14691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:17:43.992164 14691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "f980356cc0f44aac9bf489701d71af0e"
format_stamp: "Formatted at 2026-09-29 02:17:43 on dist-test-slave-9k0h"
server_key: "4fbc210b3b2348d3ac980563f927e3c7"
server_key_iv: "2cb16efae74070ffa39137e2149f23dd"
server_key_version: "encryptionkey@0"
I20260929 02:17:43.992352 14691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:44.028906 14691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:44.032891 14691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:44.033190 14691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:44.051254 14691 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:40749
I20260929 02:17:44.051265 14757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:40749 every 8 connection(s)
I20260929 02:17:44.052551 14691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:44.056016 14758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:44.062060 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 14691
I20260929 02:17:44.062273 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:44.062839 25442 external_mini_cluster.cc:1428] Setting key 65960b21110962f986b22f49d30dc9ed
I20260929 02:17:44.066433 14758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e: Bootstrap starting.
I20260929 02:17:44.069888 14758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:44.071005 14758 log.cc:824] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:44.073840 14758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e: No bootstrap required, opened a new log
Sep 29 02:17:44 dist-test-slave-9k0h krb5kdc[14678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:17:44.081557 14758 raft_consensus.cc:348] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f980356cc0f44aac9bf489701d71af0e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 40749 } }
I20260929 02:17:44.081928 14758 raft_consensus.cc:374] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:44.082038 14758 raft_consensus.cc:729] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f980356cc0f44aac9bf489701d71af0e, State: Initialized, Role: FOLLOWER
I20260929 02:17:44.082623 14758 consensus_queue.cc:260] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f980356cc0f44aac9bf489701d71af0e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 40749 } }
I20260929 02:17:44.082859 14758 raft_consensus.cc:388] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:44.083024 14758 raft_consensus.cc:482] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:44.083200 14758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:44.084440 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:44.064531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39705 (local address 127.24.216.190:40749)
0929 02:17:44.065250 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:44.065260 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:44.065297 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:17:44.066952 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:44.066973 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:44.067312 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:44.067608 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:44.069391 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:44.070842 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:44.072411 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:44.073036 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:44.073183 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:44.077160 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:44.077167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:44.077180 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:44.077225 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:44.080232 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:44.080973 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:44.080979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:44.080985 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:44.081138 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:44.081579 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:44.081582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:44.081584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:44.082195 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:44.082521 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:44.083105 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 02:17:44.083428 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":349,"threads_started":1}
I20260929 02:17:44.085721 14758 raft_consensus.cc:504] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f980356cc0f44aac9bf489701d71af0e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 40749 } }
I20260929 02:17:44.086282 14758 leader_election.cc:302] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f980356cc0f44aac9bf489701d71af0e; no voters:
I20260929 02:17:44.086979 14758 leader_election.cc:288] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:44.087044 14763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:44.088565 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:44.089008 14758 sys_catalog.cc:564] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:44.093742 14763 raft_consensus.cc:686] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [term 1 LEADER]: Becoming Leader. State: Replica: f980356cc0f44aac9bf489701d71af0e, State: Running, Role: LEADER
I20260929 02:17:44.094304 14763 consensus_queue.cc:237] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f980356cc0f44aac9bf489701d71af0e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 40749 } }
I20260929 02:17:44.097661 14764 sys_catalog.cc:455] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f980356cc0f44aac9bf489701d71af0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f980356cc0f44aac9bf489701d71af0e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 40749 } } }
I20260929 02:17:44.098086 14764 sys_catalog.cc:458] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:44.099695 14765 sys_catalog.cc:455] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f980356cc0f44aac9bf489701d71af0e. Latest consensus state: current_term: 1 leader_uuid: "f980356cc0f44aac9bf489701d71af0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f980356cc0f44aac9bf489701d71af0e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 40749 } } }
I20260929 02:17:44.100036 14765 sys_catalog.cc:458] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:44.108678 14777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:44.117622 14777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:44.121023 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:44.096432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40217 (local address 127.24.216.190:40749)
0929 02:17:44.096619 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:44.096628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:44.097139 (+ 511us) server_negotiation.cc:407] Connection header received
0929 02:17:44.097355 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:44.097365 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:44.097475 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:44.097617 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:44.100267 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:44.101871 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:44.104540 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:44.105502 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:44.105636 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:44.110479 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:44.110511 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:44.110525 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:44.110604 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:44.115756 (+ 5152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:44.117063 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:44.117074 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:44.117078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:44.117235 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:44.118060 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:44.118069 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:44.118075 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:44.118700 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:44.119063 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:44.120087 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0929 02:17:44.120370 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 02:17:44.127857 14777 catalog_manager.cc:1347] Generated new cluster ID: a5a2a4d789084616b0064cf6fdbb0537
I20260929 02:17:44.127987 14777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:44.142170 14777 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:44.143968 14777 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:44.155227 14777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f980356cc0f44aac9bf489701d71af0e: Generated new TSK 0
I20260929 02:17:44.156327 14777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:44.167943 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14691
I20260929 02:17:44.169334 14753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:44.258534 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 14691
W20260929 02:17:44.275784 14676 connection.cc:535] client connection to 127.24.216.190:40749 recv error: Network error: failed to read from TLS socket (remote: 127.24.216.190:40749): Connection reset by peer (error 104)
2026-09-29T02:17:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2046 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:44 dist-test-slave-9k0h krb5kdc[14790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:44 dist-test-slave-9k0h krb5kdc[14790](info): set up 2 sockets
Sep 29 02:17:44 dist-test-slave-9k0h krb5kdc[14790](info): commencing operation
krb5kdc: starting...
W20260929 02:17:45.366665 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:45 dist-test-slave-9k0h krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:17:45.636833 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:37109
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:37109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:45.959786 14803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:45.960043 14803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:45.960232 14803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:45.967494 14803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:45.967612 14803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:45.967656 14803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:45.967743 14803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:45.967808 14803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:45.977339 14803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:37109
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:37109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:45.978607 14803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:45.979900 14803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:45.991503 14808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:45.991638 14811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:45.991593 14809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:45.992661 14803 server_base.cc:1034] running on GCE node
I20260929 02:17:45.993429 14803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:45.994845 14803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:45.996066 14803 hybrid_clock.cc:648] HybridClock initialized: now 1790648265996035 us; error 55 us; skew 500 ppm
I20260929 02:17:45.996392 14803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:45 dist-test-slave-9k0h krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648265, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:45.999461 14803 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:17:46.000766 14803 webserver.cc:460] Webserver started at http://127.24.216.190:42019/ using document root <none> and password file <none>
I20260929 02:17:46.001353 14803 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:46.001461 14803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:46.001694 14803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:46.004451 14803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "1f10436e1e2e470fab374cd005a9858e"
format_stamp: "Formatted at 2026-09-29 02:17:45 on dist-test-slave-9k0h"
server_key: "1da243c41a5b2202850dfb099dcb4664"
server_key_iv: "bb6729944af09e12d68ca4c64eae8b80"
server_key_version: "encryptionkey@0"
I20260929 02:17:46.005136 14803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "1f10436e1e2e470fab374cd005a9858e"
format_stamp: "Formatted at 2026-09-29 02:17:45 on dist-test-slave-9k0h"
server_key: "1da243c41a5b2202850dfb099dcb4664"
server_key_iv: "bb6729944af09e12d68ca4c64eae8b80"
server_key_version: "encryptionkey@0"
I20260929 02:17:46.009816 14803 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260929 02:17:46.013149 14818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:46.014467 14803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:17:46.014607 14803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "1f10436e1e2e470fab374cd005a9858e"
format_stamp: "Formatted at 2026-09-29 02:17:45 on dist-test-slave-9k0h"
server_key: "1da243c41a5b2202850dfb099dcb4664"
server_key_iv: "bb6729944af09e12d68ca4c64eae8b80"
server_key_version: "encryptionkey@0"
I20260929 02:17:46.014762 14803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:46.038599 14803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:46.042809 14803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:46.043033 14803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:46.061431 14803 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:37109
I20260929 02:17:46.061460 14869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:37109 every 8 connection(s)
I20260929 02:17:46.062816 14803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:46.063078 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 14803
I20260929 02:17:46.063251 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:46.063750 25442 external_mini_cluster.cc:1428] Setting key 378869ee30710828af27d123b7e16c4e
I20260929 02:17:46.066998 14870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:46.076301 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:46.065508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48367 (local address 127.24.216.190:37109)
0929 02:17:46.066305 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:46.066315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:46.066355 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:17:46.068018 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:46.068038 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:46.068383 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:46.068741 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:46.069262 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:46.070627 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:46.071966 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:46.072597 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:46.072788 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:46.074039 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:46.074046 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:46.074087 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:46.074744 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:46.074820 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:46.074846 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:46.075019 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:46.075294 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 02:17:46.075456 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":355,"threads_started":1}
I20260929 02:17:46.077261 14870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e: Bootstrap starting.
I20260929 02:17:46.079952 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:46.080749 14870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:46.081655 14870 log.cc:824] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:46.084770 14870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e: No bootstrap required, opened a new log
I20260929 02:17:46.091207 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:46.084390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57279 (local address 127.24.216.190:37109)
0929 02:17:46.084575 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:46.084584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:46.085059 (+ 475us) server_negotiation.cc:407] Connection header received
0929 02:17:46.085292 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:46.085300 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:46.085356 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:46.085492 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:46.086051 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:46.087008 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:46.088365 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:46.088910 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:46.089086 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:46.090318 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:46.090324 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:46.090327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:46.090348 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:46.090401 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:46.090421 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:46.090577 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:46.090812 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 02:17:46.090877 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:17:46.092638 14870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10436e1e2e470fab374cd005a9858e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37109 } }
I20260929 02:17:46.093093 14870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:46.093214 14870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10436e1e2e470fab374cd005a9858e, State: Initialized, Role: FOLLOWER
I20260929 02:17:46.093743 14870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10436e1e2e470fab374cd005a9858e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37109 } }
I20260929 02:17:46.093955 14870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:46.094087 14870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:46.094281 14870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:46.096302 14870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10436e1e2e470fab374cd005a9858e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37109 } }
I20260929 02:17:46.096716 14870 leader_election.cc:302] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f10436e1e2e470fab374cd005a9858e; no voters:
I20260929 02:17:46.097270 14870 leader_election.cc:288] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:46.097429 14880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:46.098658 14880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [term 1 LEADER]: Becoming Leader. State: Replica: 1f10436e1e2e470fab374cd005a9858e, State: Running, Role: LEADER
I20260929 02:17:46.099313 14880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10436e1e2e470fab374cd005a9858e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37109 } }
I20260929 02:17:46.099507 14870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:46.102552 14881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f10436e1e2e470fab374cd005a9858e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10436e1e2e470fab374cd005a9858e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37109 } } }
I20260929 02:17:46.102492 14882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f10436e1e2e470fab374cd005a9858e. Latest consensus state: current_term: 1 leader_uuid: "1f10436e1e2e470fab374cd005a9858e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f10436e1e2e470fab374cd005a9858e" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37109 } } }
I20260929 02:17:46.103009 14881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:46.103008 14882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:46.103915 14887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:46.109711 14887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:46.121028 14887 catalog_manager.cc:1347] Generated new cluster ID: 44e1b77fb339435e82d39da315b866f1
I20260929 02:17:46.121168 14887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:46.143939 14887 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:46.145171 14887 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:46.156848 14887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f10436e1e2e470fab374cd005a9858e: Generated new TSK 0
I20260929 02:17:46.157766 14887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:46.175966 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14803
W20260929 02:17:46.177855 14865 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48367
W20260929 02:17:46.178611 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14803: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:17:46.178771 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 14803
W20260929 02:17:46.196108 14788 connection.cc:535] client connection to 127.24.216.190:37109 recv error: Network error: failed to read from TLS socket (remote: 127.24.216.190:37109): Connection reset by peer (error 104)
2026-09-29T02:17:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1920 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:46 dist-test-slave-9k0h krb5kdc[14901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:46 dist-test-slave-9k0h krb5kdc[14901](info): set up 2 sockets
Sep 29 02:17:46 dist-test-slave-9k0h krb5kdc[14901](info): commencing operation
krb5kdc: starting...
W20260929 02:17:47.290699 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:47 dist-test-slave-9k0h krb5kdc[14901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:17:47.561533 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:37263
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:37263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:47.882257 14914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:47.882525 14914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:47.882688 14914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:47.889767 14914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:47.889869 14914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:47.889971 14914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:47.890077 14914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:47.890174 14914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:47.903851 14914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:37263
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:37263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:47.905149 14914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:47.906862 14914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:47.918375 14920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:47.918751 14919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:47.918457 14922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:47.919250 14914 server_base.cc:1034] running on GCE node
I20260929 02:17:47.920013 14914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:47.921341 14914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:47.922603 14914 hybrid_clock.cc:648] HybridClock initialized: now 1790648267922569 us; error 44 us; skew 500 ppm
I20260929 02:17:47.922971 14914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:47 dist-test-slave-9k0h krb5kdc[14901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648267, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:47.926067 14914 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:17:47.927510 14914 webserver.cc:460] Webserver started at http://127.24.216.190:43411/ using document root <none> and password file <none>
I20260929 02:17:47.928110 14914 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:47.928215 14914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:47.928458 14914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:47.931317 14914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "25570218fca343489edd4dbf5ba2ca71"
format_stamp: "Formatted at 2026-09-29 02:17:47 on dist-test-slave-9k0h"
server_key: "14c96588a5b6e5bc0278d4509e39d29d"
server_key_iv: "39699bfb030a08476cad4e6487985cf2"
server_key_version: "encryptionkey@0"
I20260929 02:17:47.931984 14914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "25570218fca343489edd4dbf5ba2ca71"
format_stamp: "Formatted at 2026-09-29 02:17:47 on dist-test-slave-9k0h"
server_key: "14c96588a5b6e5bc0278d4509e39d29d"
server_key_iv: "39699bfb030a08476cad4e6487985cf2"
server_key_version: "encryptionkey@0"
I20260929 02:17:47.936846 14914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:17:47.940167 14929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:47.941408 14914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:17:47.941577 14914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "25570218fca343489edd4dbf5ba2ca71"
format_stamp: "Formatted at 2026-09-29 02:17:47 on dist-test-slave-9k0h"
server_key: "14c96588a5b6e5bc0278d4509e39d29d"
server_key_iv: "39699bfb030a08476cad4e6487985cf2"
server_key_version: "encryptionkey@0"
I20260929 02:17:47.941766 14914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:47.947335 14914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:47.951426 14914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:47.951716 14914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:47.969611 14914 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:37263
I20260929 02:17:47.969646 14980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:37263 every 8 connection(s)
I20260929 02:17:47.971056 14914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:47.974766 14981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:47.977550 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 14914
I20260929 02:17:47.977741 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:47.978355 25442 external_mini_cluster.cc:1428] Setting key 3ee34fa28f9ccf962852fe7ab413f8b7
I20260929 02:17:47.985913 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:47.980102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37201 (local address 127.24.216.190:37263)
0929 02:17:47.980883 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:47.980893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:47.980932 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:17:47.982592 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:47.982614 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:47.982935 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:47.983262 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:47.984354 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:47.984369 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:47.984381 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:47.984399 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:47.984469 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:47.984499 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:47.984699 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:47.984802 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 02:17:47.985017 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":370,"threads_started":1}
I20260929 02:17:47.986732 14981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71: Bootstrap starting.
I20260929 02:17:47.989382 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:47.989886 14981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:47.991087 14981 log.cc:824] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:47.994283 14981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71: No bootstrap required, opened a new log
I20260929 02:17:47.996861 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:47.994479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49089 (local address 127.24.216.190:37263)
0929 02:17:47.994648 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:47.994654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:47.995049 (+ 395us) server_negotiation.cc:407] Connection header received
0929 02:17:47.995180 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:47.995185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:47.995241 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:47.995343 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:47.996228 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:47.996239 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:47.996243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:47.996260 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:47.996309 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:47.996328 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:47.996447 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:47.996520 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 02:17:47.996573 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:17:48.002349 14981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25570218fca343489edd4dbf5ba2ca71" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37263 } }
I20260929 02:17:48.002671 14981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:48.002741 14981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25570218fca343489edd4dbf5ba2ca71, State: Initialized, Role: FOLLOWER
I20260929 02:17:48.003197 14981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25570218fca343489edd4dbf5ba2ca71" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37263 } }
I20260929 02:17:48.003379 14981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:48.003471 14981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:48.003588 14981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:48.005556 14981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25570218fca343489edd4dbf5ba2ca71" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37263 } }
I20260929 02:17:48.006027 14981 leader_election.cc:302] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25570218fca343489edd4dbf5ba2ca71; no voters:
I20260929 02:17:48.006755 14981 leader_election.cc:288] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:48.006850 14991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:48.007470 14991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [term 1 LEADER]: Becoming Leader. State: Replica: 25570218fca343489edd4dbf5ba2ca71, State: Running, Role: LEADER
I20260929 02:17:48.008054 14991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25570218fca343489edd4dbf5ba2ca71" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37263 } }
I20260929 02:17:48.008873 14981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:48.011646 14992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25570218fca343489edd4dbf5ba2ca71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25570218fca343489edd4dbf5ba2ca71" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37263 } } }
I20260929 02:17:48.012141 14992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:48.012136 14993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25570218fca343489edd4dbf5ba2ca71. Latest consensus state: current_term: 1 leader_uuid: "25570218fca343489edd4dbf5ba2ca71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25570218fca343489edd4dbf5ba2ca71" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 37263 } } }
I20260929 02:17:48.012821 14993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:48.015260 15000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:48.020740 15000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:48.028422 15000 catalog_manager.cc:1347] Generated new cluster ID: 62bbe75ed371470eafafe557eef61605
I20260929 02:17:48.028530 15000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:48.038503 15000 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:48.038898 15000 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:48.047600 15000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25570218fca343489edd4dbf5ba2ca71: Generated new TSK 0
I20260929 02:17:48.048337 15000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:48.058765 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14914
W20260929 02:17:48.059839 14976 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37201
W20260929 02:17:48.060586 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 14914: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:17:48.060748 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 14914
2026-09-29T02:17:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1881 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:48 dist-test-slave-9k0h krb5kdc[15012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:48 dist-test-slave-9k0h krb5kdc[15012](info): set up 2 sockets
Sep 29 02:17:48 dist-test-slave-9k0h krb5kdc[15012](info): commencing operation
krb5kdc: starting...
W20260929 02:17:49.166664 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.007s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:49 dist-test-slave-9k0h krb5kdc[15012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:17:49.437427 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:45879
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:45879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:49.756711 15025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:49.756960 15025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:49.757136 15025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:49.764658 15025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:49.764751 15025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:49.764854 15025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:49.764936 15025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:49.765012 15025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:49.774276 15025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:45879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:45879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:49.775454 15025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:49.776787 15025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:49.788112 15030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:49.788067 15031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:49.788017 15033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:49.789116 15025 server_base.cc:1034] running on GCE node
I20260929 02:17:49.789829 15025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:49.791173 15025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:49.792412 15025 hybrid_clock.cc:648] HybridClock initialized: now 1790648269792367 us; error 40 us; skew 500 ppm
I20260929 02:17:49.792763 15025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:49 dist-test-slave-9k0h krb5kdc[15012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648269, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:49.795915 15025 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:17:49.797242 15025 webserver.cc:460] Webserver started at http://127.24.216.190:42853/ using document root <none> and password file <none>
I20260929 02:17:49.797873 15025 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:49.798012 15025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:49.798358 15025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:49.801162 15025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "756055fbec034ed098db24846007f0cd"
format_stamp: "Formatted at 2026-09-29 02:17:49 on dist-test-slave-9k0h"
server_key: "f5ba19f6127b7aa8c65b97e2c4053f62"
server_key_iv: "92245d35ac0cf00dae3cc458f7bb4c38"
server_key_version: "encryptionkey@0"
I20260929 02:17:49.801887 15025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "756055fbec034ed098db24846007f0cd"
format_stamp: "Formatted at 2026-09-29 02:17:49 on dist-test-slave-9k0h"
server_key: "f5ba19f6127b7aa8c65b97e2c4053f62"
server_key_iv: "92245d35ac0cf00dae3cc458f7bb4c38"
server_key_version: "encryptionkey@0"
I20260929 02:17:49.806566 15025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:17:49.809634 15040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:49.810901 15025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 02:17:49.811070 15025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "756055fbec034ed098db24846007f0cd"
format_stamp: "Formatted at 2026-09-29 02:17:49 on dist-test-slave-9k0h"
server_key: "f5ba19f6127b7aa8c65b97e2c4053f62"
server_key_iv: "92245d35ac0cf00dae3cc458f7bb4c38"
server_key_version: "encryptionkey@0"
I20260929 02:17:49.811254 15025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:49.837766 15025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:49.841969 15025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:49.842319 15025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:49.860275 15025 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:45879
I20260929 02:17:49.860306 15091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:45879 every 8 connection(s)
I20260929 02:17:49.861645 15025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:49.865058 15092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:49.871588 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 15025
I20260929 02:17:49.871783 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:49.872197 25442 external_mini_cluster.cc:1428] Setting key df9033dc38515082ec71bdc8ee2f1548
I20260929 02:17:49.875360 15092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd: Bootstrap starting.
I20260929 02:17:49.878857 15092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:49.880205 15092 log.cc:824] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:49.883128 15092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd: No bootstrap required, opened a new log
Sep 29 02:17:49 dist-test-slave-9k0h krb5kdc[15012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:17:49.892071 15092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756055fbec034ed098db24846007f0cd" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45879 } }
I20260929 02:17:49.892488 15092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:49.892594 15092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 756055fbec034ed098db24846007f0cd, State: Initialized, Role: FOLLOWER
I20260929 02:17:49.893261 15092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756055fbec034ed098db24846007f0cd" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45879 } }
I20260929 02:17:49.893563 15092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:49.893720 15092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:49.893909 15092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:49.895143 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:49.873867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59253 (local address 127.24.216.190:45879)
0929 02:17:49.874651 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:49.874662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:49.874699 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:17:49.876382 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:49.876401 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:49.876750 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:49.877042 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:49.878937 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:49.880950 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:49.882376 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:49.883379 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:49.883580 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:49.887795 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:49.887802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:49.887814 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:49.887846 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:49.890911 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:49.891699 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:49.891705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:49.891710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:49.891858 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:49.892254 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:49.892257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:49.892259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:49.892873 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:49.893167 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:49.893595 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 02:17:49.894067 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":382,"threads_started":1}
I20260929 02:17:49.896427 15092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756055fbec034ed098db24846007f0cd" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45879 } }
I20260929 02:17:49.896934 15092 leader_election.cc:302] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 756055fbec034ed098db24846007f0cd; no voters:
I20260929 02:17:49.897579 15092 leader_election.cc:288] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:49.897686 15097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:49.898813 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:49.898763 15097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [term 1 LEADER]: Becoming Leader. State: Replica: 756055fbec034ed098db24846007f0cd, State: Running, Role: LEADER
I20260929 02:17:49.899314 15097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756055fbec034ed098db24846007f0cd" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45879 } }
I20260929 02:17:49.899847 15092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:49.902472 15099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 756055fbec034ed098db24846007f0cd. Latest consensus state: current_term: 1 leader_uuid: "756055fbec034ed098db24846007f0cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756055fbec034ed098db24846007f0cd" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45879 } } }
I20260929 02:17:49.903090 15099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:49.902815 15098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "756055fbec034ed098db24846007f0cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756055fbec034ed098db24846007f0cd" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 45879 } } }
I20260929 02:17:49.909052 15098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:49.910986 15106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:49.917208 15106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:49.928488 15106 catalog_manager.cc:1347] Generated new cluster ID: aff21bee05164b329942ed549ba75e85
I20260929 02:17:49.928619 15106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:49.946020 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:49.914643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50123 (local address 127.24.216.190:45879)
0929 02:17:49.914817 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:49.914826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:49.927820 (+ 12994us) server_negotiation.cc:407] Connection header received
0929 02:17:49.928268 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:49.928274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:49.928350 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:49.928425 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:49.930485 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:49.931655 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:49.935790 (+ 4135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:49.936439 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:49.936529 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:49.939503 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:49.939509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:49.939514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:49.939547 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:49.943070 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:49.943827 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:49.943833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:49.943836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:49.943982 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:49.944376 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:49.944382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:49.944385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:49.944746 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:49.944960 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:49.945464 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 02:17:49.945658 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":53}
I20260929 02:17:49.953765 15106 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:49.955399 15106 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:49.961730 15106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 756055fbec034ed098db24846007f0cd: Generated new TSK 0
I20260929 02:17:49.962553 15106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:49.970204 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 15025
I20260929 02:17:49.971217 15087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:50.058480 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 15025
2026-09-29T02:17:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:50 dist-test-slave-9k0h krb5kdc[15124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:50 dist-test-slave-9k0h krb5kdc[15124](info): set up 2 sockets
Sep 29 02:17:50 dist-test-slave-9k0h krb5kdc[15124](info): commencing operation
krb5kdc: starting...
W20260929 02:17:51.154559 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:51 dist-test-slave-9k0h krb5kdc[15124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:17:51.423485 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:39193
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:39193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:51.754212 15137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:51.754608 15137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:51.754765 15137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:51.761916 15137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:51.762070 15137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:51.762193 15137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:51.762305 15137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:51.762391 15137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:51.771940 15137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:39193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:39193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:51.773229 15137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:51.774576 15137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:51.786257 15142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:51.786262 15145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:51.786247 15143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:51.787129 15137 server_base.cc:1034] running on GCE node
I20260929 02:17:51.787904 15137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:51.789291 15137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:51.790509 15137 hybrid_clock.cc:648] HybridClock initialized: now 1790648271790490 us; error 62 us; skew 500 ppm
I20260929 02:17:51.790830 15137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:51 dist-test-slave-9k0h krb5kdc[15124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648271, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:51.793741 15137 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:17:51.795259 15137 webserver.cc:460] Webserver started at http://127.24.216.190:37473/ using document root <none> and password file <none>
I20260929 02:17:51.795907 15137 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:51.796049 15137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:51.796352 15137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:51.799281 15137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "b21fc0c990ae40a3b4ea1689a2942aa7"
format_stamp: "Formatted at 2026-09-29 02:17:51 on dist-test-slave-9k0h"
server_key: "a444e40369acd6d304f02944dec9ba7d"
server_key_iv: "353668ea727abaf2934f629061ac4327"
server_key_version: "encryptionkey@0"
I20260929 02:17:51.800040 15137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "b21fc0c990ae40a3b4ea1689a2942aa7"
format_stamp: "Formatted at 2026-09-29 02:17:51 on dist-test-slave-9k0h"
server_key: "a444e40369acd6d304f02944dec9ba7d"
server_key_iv: "353668ea727abaf2934f629061ac4327"
server_key_version: "encryptionkey@0"
I20260929 02:17:51.804998 15137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:17:51.808341 15152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:51.809574 15137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:17:51.809758 15137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "b21fc0c990ae40a3b4ea1689a2942aa7"
format_stamp: "Formatted at 2026-09-29 02:17:51 on dist-test-slave-9k0h"
server_key: "a444e40369acd6d304f02944dec9ba7d"
server_key_iv: "353668ea727abaf2934f629061ac4327"
server_key_version: "encryptionkey@0"
I20260929 02:17:51.809953 15137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:51.831580 15137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:51.835783 15137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:51.836007 15137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:51.852823 15137 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:39193
I20260929 02:17:51.852846 15203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:39193 every 8 connection(s)
I20260929 02:17:51.854557 15137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:51.857720 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 15137
I20260929 02:17:51.857903 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:51.858412 25442 external_mini_cluster.cc:1428] Setting key 8e6ece294386fcf92eda036ef4e39057
I20260929 02:17:51.858438 15204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:51.870249 15204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7: Bootstrap starting.
I20260929 02:17:51.871109 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:51.860094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34745 (local address 127.24.216.190:39193)
0929 02:17:51.860868 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:51.860879 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:51.860923 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:17:51.862953 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:51.862974 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:51.863340 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:51.863739 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:51.864278 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:51.865565 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:51.866919 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:51.867596 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:51.867786 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:51.868953 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:51.868960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:51.868973 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:51.869461 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:51.869529 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:51.869552 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:51.869682 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:51.869913 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 02:17:51.870191 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":345,"threads_started":1}
I20260929 02:17:51.873258 15204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:51.874442 15204 log.cc:824] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:51.874614 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:51.877154 15204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7: No bootstrap required, opened a new log
I20260929 02:17:51.884757 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:51.878867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37395 (local address 127.24.216.190:39193)
0929 02:17:51.879061 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:51.879069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:51.879435 (+ 366us) server_negotiation.cc:407] Connection header received
0929 02:17:51.879650 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:51.879654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:51.879711 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:51.879835 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:51.880320 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:51.881036 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:51.882397 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:51.882867 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:51.883010 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:51.884062 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:51.884066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:51.884070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:51.884087 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:51.884121 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:51.884138 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:51.884243 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:51.884376 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 02:17:51.884459 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:17:51.885118 15204 raft_consensus.cc:348] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21fc0c990ae40a3b4ea1689a2942aa7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39193 } }
I20260929 02:17:51.885500 15204 raft_consensus.cc:374] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:51.885617 15204 raft_consensus.cc:729] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21fc0c990ae40a3b4ea1689a2942aa7, State: Initialized, Role: FOLLOWER
I20260929 02:17:51.886242 15204 consensus_queue.cc:260] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21fc0c990ae40a3b4ea1689a2942aa7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39193 } }
I20260929 02:17:51.886526 15204 raft_consensus.cc:388] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:51.886672 15204 raft_consensus.cc:482] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:51.886852 15204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:51.889282 15204 raft_consensus.cc:504] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21fc0c990ae40a3b4ea1689a2942aa7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39193 } }
I20260929 02:17:51.889765 15204 leader_election.cc:302] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b21fc0c990ae40a3b4ea1689a2942aa7; no voters:
I20260929 02:17:51.890364 15204 leader_election.cc:288] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:51.890479 15214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:51.891191 15214 raft_consensus.cc:686] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [term 1 LEADER]: Becoming Leader. State: Replica: b21fc0c990ae40a3b4ea1689a2942aa7, State: Running, Role: LEADER
I20260929 02:17:51.891719 15214 consensus_queue.cc:237] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21fc0c990ae40a3b4ea1689a2942aa7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39193 } }
I20260929 02:17:51.892418 15204 sys_catalog.cc:564] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:51.894415 15215 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b21fc0c990ae40a3b4ea1689a2942aa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21fc0c990ae40a3b4ea1689a2942aa7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39193 } } }
I20260929 02:17:51.894472 15216 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b21fc0c990ae40a3b4ea1689a2942aa7. Latest consensus state: current_term: 1 leader_uuid: "b21fc0c990ae40a3b4ea1689a2942aa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21fc0c990ae40a3b4ea1689a2942aa7" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 39193 } } }
I20260929 02:17:51.894874 15216 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:51.894874 15215 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:51.896817 15220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:51.902002 15220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:51.910656 15220 catalog_manager.cc:1347] Generated new cluster ID: ac6225af9e014b87b728e2791be89452
I20260929 02:17:51.910778 15220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:51.928673 15220 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:51.930310 15220 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:51.945063 15220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b21fc0c990ae40a3b4ea1689a2942aa7: Generated new TSK 0
I20260929 02:17:51.945888 15220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:51.956207 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 15137
W20260929 02:17:51.957201 15199 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34745
W20260929 02:17:51.957894 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 15137: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:17:51.958041 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 15137
2026-09-29T02:17:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1898 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:52 dist-test-slave-9k0h krb5kdc[15235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:52 dist-test-slave-9k0h krb5kdc[15235](info): set up 2 sockets
Sep 29 02:17:52 dist-test-slave-9k0h krb5kdc[15235](info): commencing operation
krb5kdc: starting...
W20260929 02:17:53.082720 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:53 dist-test-slave-9k0h krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:17:53.354786 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:35687
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:40575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:35687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:53.680332 15248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:53.680611 15248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:53.680702 15248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:53.687884 15248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:53.687994 15248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:53.688050 15248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:53.688108 15248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:53.688153 15248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:53.697693 15248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:40575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:35687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:35687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:53.698994 15248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:53.700232 15248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:53.711987 15253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:53.711970 15256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:53.712010 15254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:53.712816 15248 server_base.cc:1034] running on GCE node
I20260929 02:17:53.713563 15248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:53.714996 15248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:53.716207 15248 hybrid_clock.cc:648] HybridClock initialized: now 1790648273716171 us; error 33 us; skew 500 ppm
I20260929 02:17:53.716554 15248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:53 dist-test-slave-9k0h krb5kdc[15235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648273, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:53.719602 15248 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:17:53.720970 15248 webserver.cc:460] Webserver started at http://127.24.216.190:40937/ using document root <none> and password file <none>
I20260929 02:17:53.721640 15248 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:53.721783 15248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:53.722079 15248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:53.724956 15248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "528067743cd7416b841a74d123743815"
format_stamp: "Formatted at 2026-09-29 02:17:53 on dist-test-slave-9k0h"
server_key: "dafeedb379bab4bb43f95b249e6368e2"
server_key_iv: "8e013eba3e206244afa2883f16251440"
server_key_version: "encryptionkey@0"
I20260929 02:17:53.725745 15248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "528067743cd7416b841a74d123743815"
format_stamp: "Formatted at 2026-09-29 02:17:53 on dist-test-slave-9k0h"
server_key: "dafeedb379bab4bb43f95b249e6368e2"
server_key_iv: "8e013eba3e206244afa2883f16251440"
server_key_version: "encryptionkey@0"
I20260929 02:17:53.730671 15248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:17:53.733898 15263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:53.735164 15248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:17:53.735345 15248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "528067743cd7416b841a74d123743815"
format_stamp: "Formatted at 2026-09-29 02:17:53 on dist-test-slave-9k0h"
server_key: "dafeedb379bab4bb43f95b249e6368e2"
server_key_iv: "8e013eba3e206244afa2883f16251440"
server_key_version: "encryptionkey@0"
I20260929 02:17:53.735538 15248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:53.741115 15248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:53.745137 15248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:53.745422 15248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:53.763597 15248 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:35687
I20260929 02:17:53.763633 15314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:35687 every 8 connection(s)
I20260929 02:17:53.764957 15248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:53.768680 15315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:53.769481 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 15248
I20260929 02:17:53.769655 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:53.770093 25442 external_mini_cluster.cc:1428] Setting key f0d4c79953909e9169d3710eb44942c8
I20260929 02:17:53.778962 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:53.771819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52445 (local address 127.24.216.190:35687)
0929 02:17:53.772590 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:53.772605 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:53.772657 (+ 52us) server_negotiation.cc:407] Connection header received
0929 02:17:53.775250 (+ 2593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:53.775279 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:53.775704 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:53.776148 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:53.777076 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:53.777084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:53.777100 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:53.777117 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:53.777178 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:53.777201 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:53.777399 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:53.777569 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 02:17:53.777749 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":315,"threads_started":1}
I20260929 02:17:53.782754 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:53.783252 15315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815: Bootstrap starting.
I20260929 02:17:53.786284 15315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:53.787616 15315 log.cc:824] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:53.789678 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:53.787162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56813 (local address 127.24.216.190:35687)
0929 02:17:53.787346 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:53.787353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:53.787734 (+ 381us) server_negotiation.cc:407] Connection header received
0929 02:17:53.787935 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:53.787940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:53.787986 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:53.788090 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:53.789013 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:53.789018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:53.789023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:17:53.789040 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:53.789085 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:17:53.789099 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:53.789215 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:53.789346 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 02:17:53.789405 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:17:53.790596 15315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815: No bootstrap required, opened a new log
I20260929 02:17:53.798368 15315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528067743cd7416b841a74d123743815" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35687 } }
I20260929 02:17:53.798759 15315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:17:53.798875 15315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528067743cd7416b841a74d123743815, State: Initialized, Role: FOLLOWER
I20260929 02:17:53.799585 15315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528067743cd7416b841a74d123743815" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35687 } }
I20260929 02:17:53.799887 15315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:17:53.800035 15315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:17:53.800217 15315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:53.802417 15315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528067743cd7416b841a74d123743815" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35687 } }
I20260929 02:17:53.802891 15315 leader_election.cc:302] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 528067743cd7416b841a74d123743815; no voters:
I20260929 02:17:53.803483 15315 leader_election.cc:288] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:17:53.803615 15325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:53.804427 15325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [term 1 LEADER]: Becoming Leader. State: Replica: 528067743cd7416b841a74d123743815, State: Running, Role: LEADER
I20260929 02:17:53.805007 15325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528067743cd7416b841a74d123743815" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35687 } }
I20260929 02:17:53.805613 15315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:53.807747 15326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "528067743cd7416b841a74d123743815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528067743cd7416b841a74d123743815" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35687 } } }
I20260929 02:17:53.808279 15326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:53.809085 15327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 528067743cd7416b841a74d123743815. Latest consensus state: current_term: 1 leader_uuid: "528067743cd7416b841a74d123743815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528067743cd7416b841a74d123743815" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 35687 } } }
I20260929 02:17:53.809520 15327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:53.810019 15332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:17:53.815603 15332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:53.824589 15332 catalog_manager.cc:1347] Generated new cluster ID: 1c890e59930147649385de8e77613ccf
I20260929 02:17:53.824702 15332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:53.857115 15332 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:53.857561 15332 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:53.867383 15332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 528067743cd7416b841a74d123743815: Generated new TSK 0
I20260929 02:17:53.868201 15332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:53.882117 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 15248
W20260929 02:17:53.883279 15310 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52445
W20260929 02:17:53.883986 25442 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 15248: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:17:53.884191 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 15248
2026-09-29T02:17:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1927 ms)
[----------] 18 tests from AuthTokenIssuingTest (34929 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:53 dist-test-slave-9k0h krb5kdc[15346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:53 dist-test-slave-9k0h krb5kdc[15346](info): set up 2 sockets
Sep 29 02:17:53 dist-test-slave-9k0h krb5kdc[15346](info): commencing operation
krb5kdc: starting...
W20260929 02:17:54.994837 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:55 dist-test-slave-9k0h krb5kdc[15346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:17:55.265735 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:46373
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:41969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:46373,127.24.216.189:39863,127.24.216.188:36963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:17:55.596992 15359 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:17:55.597371 15359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:55.597500 15359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:55.597637 15359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:55.604905 15359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:55.605026 15359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:55.605130 15359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:55.605233 15359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:55.605326 15359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:55.614990 15359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:41969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:46373,127.24.216.189:39863,127.24.216.188:36963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:46373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:55.616250 15359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:55.617532 15359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:55.630434 15364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:55.630374 15367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:55.630473 15365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:55.631685 15359 server_base.cc:1034] running on GCE node
I20260929 02:17:55.632413 15359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:55.633744 15359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:55.634958 15359 hybrid_clock.cc:648] HybridClock initialized: now 1790648275634908 us; error 29 us; skew 500 ppm
I20260929 02:17:55.635251 15359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:55 dist-test-slave-9k0h krb5kdc[15346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648275, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:55.645045 15359 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:17:55.646632 15359 webserver.cc:460] Webserver started at http://127.24.216.190:35151/ using document root <none> and password file <none>
I20260929 02:17:55.647399 15359 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:55.647511 15359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:55.647845 15359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:55.650784 15359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "28f1b3d9da034b40ab094d045e2dbeb3"
format_stamp: "Formatted at 2026-09-29 02:17:55 on dist-test-slave-9k0h"
server_key: "ffc412118c7bd9ebadc16e07f5778e9d"
server_key_iv: "ab9281e90c1535eb7589883b80afd5d9"
server_key_version: "encryptionkey@0"
I20260929 02:17:55.651567 15359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "28f1b3d9da034b40ab094d045e2dbeb3"
format_stamp: "Formatted at 2026-09-29 02:17:55 on dist-test-slave-9k0h"
server_key: "ffc412118c7bd9ebadc16e07f5778e9d"
server_key_iv: "ab9281e90c1535eb7589883b80afd5d9"
server_key_version: "encryptionkey@0"
I20260929 02:17:55.656462 15359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:17:55.659857 15374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:55.661139 15359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:17:55.661322 15359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "28f1b3d9da034b40ab094d045e2dbeb3"
format_stamp: "Formatted at 2026-09-29 02:17:55 on dist-test-slave-9k0h"
server_key: "ffc412118c7bd9ebadc16e07f5778e9d"
server_key_iv: "ab9281e90c1535eb7589883b80afd5d9"
server_key_version: "encryptionkey@0"
I20260929 02:17:55.661525 15359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:55.683905 15359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:55.687853 15359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:55.688134 15359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:55.706504 15359 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:46373
I20260929 02:17:55.706522 15425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:46373 every 8 connection(s)
I20260929 02:17:55.707803 15359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:17:55.711374 15426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:55.713033 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 15359
I20260929 02:17:55.713238 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:17:55.713837 25442 external_mini_cluster.cc:1428] Setting key d5ee383ba651f3c187eb442ddf5da4b7
I20260929 02:17:55.717761 15426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:17:55 dist-test-slave-9k0h krb5kdc[15346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648275, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.216.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.189@KRBTEST.COM" created.
I20260929 02:17:55.743921 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:55.720937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:46373 (local address 127.0.0.1:38108)
0929 02:17:55.725848 (+ 4911us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:55.725887 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:55.727686 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:55.728462 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:55.728471 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:55.729172 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:55.729491 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:55.729508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:55.731132 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:55.731137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:55.732208 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:55.732219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:55.733139 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:55.733146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:55.733325 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:55.733978 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:55.733998 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:55.736620 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:55.739912 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:55.739927 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:55.739942 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:55.740523 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:55.740937 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:55.740945 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:55.740950 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:55.741164 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:55.741986 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:55.741992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:55.742246 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 02:17:55.742486 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 02:17:55.742876 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4679,"thread_start_us":4456,"threads_started":1}
I20260929 02:17:55.743939 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:55.720937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.24.216.190:46373)
0929 02:17:55.725553 (+ 4616us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:55.725572 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:55.726026 (+ 454us) server_negotiation.cc:407] Connection header received
0929 02:17:55.728040 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:55.728046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:55.728254 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:55.728449 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:55.729712 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:55.730982 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:55.732355 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:55.732911 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:55.733103 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:55.736859 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:55.736867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:55.736878 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:55.736905 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:55.739774 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:55.740660 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:55.740667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:55.740669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:55.740816 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:55.741284 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:55.741287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:55.741289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:55.741854 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:55.742008 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:55.742646 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 02:17:55.743006 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":4347,"thread_start_us":4127,"threads_started":1}
I20260929 02:17:55.746980 15426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:17:55.748167 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:55.747694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:39863 (local address 127.0.0.1:55758)
0929 02:17:55.747890 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:55.748016 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 02:17:55.748445 15378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
I20260929 02:17:55.750025 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:55.749586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:39863 (local address 127.0.0.1:55766)
0929 02:17:55.749755 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:55.749860 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 02:17:55.750286 15426 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:39863: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.189.keytab.
Entry for principal kudu/127.24.216.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.189.keytab.
I20260929 02:17:55.776697 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.189:39863
--webserver_interface=127.24.216.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.189.keytab
--principal=kudu/127.24.216.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:41969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:46373,127.24.216.189:39863,127.24.216.188:36963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:17:55.799643 15426 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } attempt: 1
I20260929 02:17:55.800850 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:55.800402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:39863 (local address 127.0.0.1:55770)
0929 02:17:55.800592 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:55.800719 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 02:17:55.801824 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:55.801508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:39863 (local address 127.0.0.1:55778)
0929 02:17:55.801640 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:55.801707 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:17:55.802090 15426 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:39863: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
I20260929 02:17:55.870366 15426 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } attempt: 2
I20260929 02:17:55.871557 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:55.871104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:39863 (local address 127.0.0.1:55794)
0929 02:17:55.871340 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:55.871429 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 02:17:55.872536 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:55.872182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:39863 (local address 127.0.0.1:55802)
0929 02:17:55.872327 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:55.872410 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 02:17:55.872720 15426 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:39863: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
I20260929 02:17:55.963984 15426 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } attempt: 3
I20260929 02:17:55.965109 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:55.964688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:39863 (local address 127.0.0.1:55810)
0929 02:17:55.964881 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:55.964981 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 02:17:55.966101 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:55.965739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:39863 (local address 127.0.0.1:55814)
0929 02:17:55.965889 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:55.965977 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 02:17:55.966342 15426 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:39863: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
W20260929 02:17:56.095585 15433 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:17:56.095880 15433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:56.095976 15433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:56.096131 15433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:17:56.103348 15433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:56.103442 15433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:56.103545 15433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:56.103636 15433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:56.103749 15433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:56.109619 15426 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } attempt: 4
I20260929 02:17:56.110836 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.110302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:39863 (local address 127.0.0.1:55824)
0929 02:17:56.110621 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.110707 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260929 02:17:56.111830 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.111526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:39863 (local address 127.0.0.1:55828)
0929 02:17:56.111646 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.111723 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 02:17:56.112080 15426 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:39863: Network error: Client connection negotiation failed: client connection to 127.24.216.189:39863: connect: Connection refused (error 111)
I20260929 02:17:56.114488 15433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:41969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:46373,127.24.216.189:39863,127.24.216.188:36963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.189.keytab
--principal=kudu/127.24.216.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.189:39863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.216.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:56.115742 15433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:56.117041 15433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:56.128214 15441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:56.128305 15438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:56.128275 15439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:56.129266 15433 server_base.cc:1034] running on GCE node
I20260929 02:17:56.129972 15433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:56.131319 15433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:56.132557 15433 hybrid_clock.cc:648] HybridClock initialized: now 1790648276132501 us; error 53 us; skew 500 ppm
I20260929 02:17:56.132884 15433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:17:56 dist-test-slave-9k0h krb5kdc[15346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648276, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:56.136034 15433 init.cc:377] Logged in from keytab as kudu/127.24.216.189@KRBTEST.COM (short username kudu)
I20260929 02:17:56.137429 15433 webserver.cc:460] Webserver started at http://127.24.216.189:44851/ using document root <none> and password file <none>
I20260929 02:17:56.138046 15433 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:56.138139 15433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:56.138477 15433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:56.141369 15433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/data/instance:
uuid: "df06b7847d1843ce9596ee9f8108f3d3"
format_stamp: "Formatted at 2026-09-29 02:17:56 on dist-test-slave-9k0h"
server_key: "fe2d05eff4805bb5e1a0c9d56821888a"
server_key_iv: "8af8a6a26fead452db588c9b7fb28e3f"
server_key_version: "encryptionkey@0"
I20260929 02:17:56.142077 15433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/wal/instance:
uuid: "df06b7847d1843ce9596ee9f8108f3d3"
format_stamp: "Formatted at 2026-09-29 02:17:56 on dist-test-slave-9k0h"
server_key: "fe2d05eff4805bb5e1a0c9d56821888a"
server_key_iv: "8af8a6a26fead452db588c9b7fb28e3f"
server_key_version: "encryptionkey@0"
I20260929 02:17:56.146858 15433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:17:56.150094 15448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:56.151367 15433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:17:56.151541 15433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/wal
uuid: "df06b7847d1843ce9596ee9f8108f3d3"
format_stamp: "Formatted at 2026-09-29 02:17:56 on dist-test-slave-9k0h"
server_key: "fe2d05eff4805bb5e1a0c9d56821888a"
server_key_iv: "8af8a6a26fead452db588c9b7fb28e3f"
server_key_version: "encryptionkey@0"
I20260929 02:17:56.151739 15433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:56.178853 15433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:56.183234 15433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:56.183508 15433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:56.201735 15433 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.189:39863
I20260929 02:17:56.201771 15499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.189:39863 every 8 connection(s)
I20260929 02:17:56.203116 15433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/data/info.pb
I20260929 02:17:56.206893 15500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:56.210846 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 15433
I20260929 02:17:56.211028 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-1/wal/instance
I20260929 02:17:56.211437 25442 external_mini_cluster.cc:1428] Setting key d4072fc5deaa719fcb8ae3ff420ba2a0
I20260929 02:17:56.213799 15500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:17:56 dist-test-slave-9k0h krb5kdc[15346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648276, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.189@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:17:56.236155 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.216709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38114 (local address 127.24.216.190:46373)
0929 02:17:56.216895 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:56.216900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:56.218191 (+ 1291us) server_negotiation.cc:407] Connection header received
0929 02:17:56.220052 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:56.220060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:56.220155 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:56.220267 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:56.222237 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.223271 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.224862 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.225287 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.225428 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.228811 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:56.228820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:56.228828 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:56.228873 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:56.232640 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.233427 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.233436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.233443 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.233624 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.234053 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.234060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.234066 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.234485 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:56.234674 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:56.235398 (+ 724us) server_negotiation.cc:299] Negotiation successful
0929 02:17:56.235666 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:17:56.236891 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.217225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:46373 (local address 127.0.0.1:38114)
0929 02:17:56.217982 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.218042 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:56.219780 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:56.220534 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:56.220547 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:56.221311 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:56.222037 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.222055 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.223436 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.223440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.224624 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.224637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.225515 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.225522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.225748 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.226475 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:56.226501 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:56.228588 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:56.232792 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.232799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.232812 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.233280 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.233748 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.233752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.233754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.233935 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.234661 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:56.234667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:56.235014 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 02:17:56.235267 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 02:17:56.235669 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":332,"threads_started":1}
I20260929 02:17:56.238772 15500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.216.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.188@KRBTEST.COM" created.
Sep 29 02:17:56 dist-test-slave-9k0h krb5kdc[15346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648276, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.189@KRBTEST.COM for kudu/127.24.216.189@KRBTEST.COM
I20260929 02:17:56.255502 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.239710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:39863 (local address 127.0.0.1:55836)
0929 02:17:56.239934 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.239955 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:56.240062 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:56.240995 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:56.240999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:56.241401 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:56.241537 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.241545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.243250 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.243254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.244068 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.244079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.245311 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.245315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.245457 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.246100 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:56.246116 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:56.248360 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:56.252666 (+ 4306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.252670 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.252672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.253019 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.253411 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.253414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.253416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.253543 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.254308 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:56.254315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:56.254479 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 02:17:56.254661 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 02:17:56.254988 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":1}
I20260929 02:17:56.255553 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.239891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55836 (local address 127.24.216.189:39863)
0929 02:17:56.240594 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:56.240600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:56.240625 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:17:56.240669 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:56.240675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:56.240838 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:56.240956 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:56.241695 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.243102 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.244212 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.245157 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.245298 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.248580 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:56.248595 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:56.248615 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:56.248656 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:56.252520 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.253148 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.253160 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.253166 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.253305 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.253643 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.253651 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.253657 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.254174 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:56.254344 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:56.254777 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 02:17:56.255139 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":181,"server-negotiator.queue_time_us":565,"thread_start_us":429,"threads_started":1}
I20260929 02:17:56.258117 15500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:17:56.259217 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.258776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:48738)
0929 02:17:56.258969 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.259062 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 02:17:56.259441 15451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
I20260929 02:17:56.261008 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.260680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:48742)
0929 02:17:56.260845 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.260915 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 02:17:56.261240 15500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:36963: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.188.keytab.
Entry for principal kudu/127.24.216.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.188.keytab.
I20260929 02:17:56.280073 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.188:36963
--webserver_interface=127.24.216.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.188.keytab
--principal=kudu/127.24.216.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:41969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:46373,127.24.216.189:39863,127.24.216.188:36963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:17:56.310508 15500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } attempt: 1
I20260929 02:17:56.311746 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.311192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:48744)
0929 02:17:56.311418 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.311556 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 02:17:56.312777 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.312367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:48754)
0929 02:17:56.312551 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.312639 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 02:17:56.313073 15500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:36963: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
I20260929 02:17:56.381283 15500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } attempt: 2
I20260929 02:17:56.382471 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.381916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:48770)
0929 02:17:56.382170 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.382283 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 02:17:56.383504 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.383117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:48786)
0929 02:17:56.383278 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.383359 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 02:17:56.383737 15500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:36963: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
I20260929 02:17:56.411379 15426 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } attempt: 5
Sep 29 02:17:56 dist-test-slave-9k0h krb5kdc[15346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648275, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for kudu/127.24.216.189@KRBTEST.COM
I20260929 02:17:56.424767 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.412247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.24.216.189:39863)
0929 02:17:56.412472 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:56.412478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:56.412500 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:17:56.412609 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:56.412614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:56.412684 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:56.412794 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:56.413598 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.414377 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.415530 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.415962 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.416095 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.419411 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:56.419416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:56.419419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:56.419457 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:56.422295 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.422918 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.422925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.422928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.423053 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.423409 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.423415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.423418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.423699 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:56.423847 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:56.424230 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 02:17:56.424434 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:17:56.424795 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.412118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:39863 (local address 127.0.0.1:39238)
0929 02:17:56.412318 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.412343 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:56.412482 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:56.412833 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:56.412841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:56.413256 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:56.413440 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.413450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.414516 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.414524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.415369 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.415380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.416109 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.416115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.416312 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.416986 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:56.417003 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:56.419225 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:56.422439 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.422444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.422448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.422799 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.423160 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.423165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.423168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.423302 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.423815 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:56.423820 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:56.423975 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 02:17:56.424128 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 02:17:56.424428 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 02:17:56.425931 15426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:17:56.426891 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.426542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56128)
0929 02:17:56.426693 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.426769 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 02:17:56.427721 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.427416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56144)
0929 02:17:56.427556 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.427625 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 02:17:56.427932 15426 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:36963: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
I20260929 02:17:56.474985 15500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } attempt: 3
I20260929 02:17:56.476349 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.475723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56150)
0929 02:17:56.475985 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.476119 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 02:17:56.477584 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.477044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56162)
0929 02:17:56.477311 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.477417 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260929 02:17:56.477828 15500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:36963: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
I20260929 02:17:56.479130 15426 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } attempt: 1
I20260929 02:17:56.480201 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.479626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56168)
0929 02:17:56.479876 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.479992 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 02:17:56.481252 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.480864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56174)
0929 02:17:56.481037 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.481125 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 02:17:56.481460 15426 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:36963: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
I20260929 02:17:56.549700 15426 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } attempt: 2
I20260929 02:17:56.551069 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.550463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56178)
0929 02:17:56.550721 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.550844 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 02:17:56.552139 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.551752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56182)
0929 02:17:56.551908 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.551995 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 02:17:56.552343 15426 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:36963: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
W20260929 02:17:56.619362 15507 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:17:56.619692 15507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:17:56.619834 15507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:17:56.619954 15507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 02:17:56.621057 15500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } attempt: 4
I20260929 02:17:56.622293 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.621690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56194)
0929 02:17:56.621962 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.622076 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 02:17:56.623265 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.622896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56210)
0929 02:17:56.623049 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.623135 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 02:17:56.623467 15500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:36963: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
W20260929 02:17:56.628756 15507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:17:56.628851 15507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:17:56.628954 15507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:17:56.629027 15507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:17:56.629109 15507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:17:56.639482 15507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:41969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:46373,127.24.216.189:39863,127.24.216.188:36963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.188.keytab
--principal=kudu/127.24.216.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.188:36963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.216.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:17:56.640697 15507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:17:56.641904 15507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:17:56.653910 15512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:56.653925 15515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:17:56.653875 15513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:17:56.654649 15507 server_base.cc:1034] running on GCE node
I20260929 02:17:56.655507 15507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:17:56.656782 15507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:17:56.658017 15507 hybrid_clock.cc:648] HybridClock initialized: now 1790648276657960 us; error 58 us; skew 500 ppm
I20260929 02:17:56.658380 15507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:17:56.658567 15426 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } attempt: 3
I20260929 02:17:56.659684 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.659123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56214)
0929 02:17:56.659347 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.659445 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 02:17:56.660704 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.660356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56226)
0929 02:17:56.660544 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.660615 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260929 02:17:56.660969 15426 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:36963: Network error: Client connection negotiation failed: client connection to 127.24.216.188:36963: connect: Connection refused (error 111)
Sep 29 02:17:56 dist-test-slave-9k0h krb5kdc[15346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648276, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:17:56.662024 15507 init.cc:377] Logged in from keytab as kudu/127.24.216.188@KRBTEST.COM (short username kudu)
I20260929 02:17:56.663805 15507 webserver.cc:460] Webserver started at http://127.24.216.188:46557/ using document root <none> and password file <none>
I20260929 02:17:56.664497 15507 fs_manager.cc:359] Metadata directory not provided
I20260929 02:17:56.664610 15507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:17:56.664947 15507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:17:56.667829 15507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/data/instance:
uuid: "e5aeb222c34c4f04bd13dcd136a15aaf"
format_stamp: "Formatted at 2026-09-29 02:17:56 on dist-test-slave-9k0h"
server_key: "c081c37b70c71880fea671e10a36cc17"
server_key_iv: "1a8d2c2cfe4bf99868571f4398ac3b61"
server_key_version: "encryptionkey@0"
I20260929 02:17:56.668591 15507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/wal/instance:
uuid: "e5aeb222c34c4f04bd13dcd136a15aaf"
format_stamp: "Formatted at 2026-09-29 02:17:56 on dist-test-slave-9k0h"
server_key: "c081c37b70c71880fea671e10a36cc17"
server_key_iv: "1a8d2c2cfe4bf99868571f4398ac3b61"
server_key_version: "encryptionkey@0"
I20260929 02:17:56.673405 15507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:17:56.676632 15522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:17:56.677855 15507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:17:56.678013 15507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/wal
uuid: "e5aeb222c34c4f04bd13dcd136a15aaf"
format_stamp: "Formatted at 2026-09-29 02:17:56 on dist-test-slave-9k0h"
server_key: "c081c37b70c71880fea671e10a36cc17"
server_key_iv: "1a8d2c2cfe4bf99868571f4398ac3b61"
server_key_version: "encryptionkey@0"
I20260929 02:17:56.678262 15507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:17:56.698648 15507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:17:56.702447 15507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:17:56.702656 15507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:17:56.721367 15507 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.188:36963
I20260929 02:17:56.721374 15573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.188:36963 every 8 connection(s)
I20260929 02:17:56.722759 15507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/data/info.pb
I20260929 02:17:56.726284 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 15507
I20260929 02:17:56.726461 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790647920678408-25442-0/minicluster-data/master-2/wal/instance
I20260929 02:17:56.726552 15574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:17:56.727025 25442 external_mini_cluster.cc:1428] Setting key eaabe9515aed32aad48c5bcb201ce63d
I20260929 02:17:56.735548 15574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:17:56 dist-test-slave-9k0h krb5kdc[15346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.188@KRBTEST.COM
Sep 29 02:17:56 dist-test-slave-9k0h krb5kdc[15346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648276, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.188@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:17:56.749039 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.728811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.24.216.188:36963)
0929 02:17:56.729630 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:56.729640 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:56.729682 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:17:56.731267 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:56.731286 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:56.731752 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:56.732176 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:56.733713 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.735026 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.736262 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.737088 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.737328 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.741951 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:56.741959 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:56.741968 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:56.742003 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:56.745200 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.745931 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.745940 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.745947 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.746138 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.746565 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.746569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.746570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.747017 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:56.747320 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:56.747773 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 02:17:56.748139 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":389,"threads_started":1}
I20260929 02:17:56.753252 15578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.738030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:46373 (local address 127.0.0.1:49274)
0929 02:17:56.738812 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.738857 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:56.738985 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:56.739724 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:56.739742 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:56.740355 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:56.740706 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.740719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.741806 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.741811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.743000 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.743011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.743766 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.743777 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.743937 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.744478 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:56.744494 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:56.746878 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:56.750052 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.750062 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.750069 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.750640 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.751009 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.751014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.751018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.751155 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.752212 (+ 1057us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:56.752218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:56.752359 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 02:17:56.752532 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 02:17:56.752838 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":428,"threads_started":1}
I20260929 02:17:56.753273 15579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.738285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.24.216.190:46373)
0929 02:17:56.739406 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:56.739410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:56.739425 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:17:56.739471 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:56.739475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:56.739528 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:56.739648 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:56.740861 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.741650 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.743145 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.743565 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.743704 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.747060 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:56.747065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:56.747068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:56.747098 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:56.749918 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.750768 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.750778 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.750780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.750897 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.751552 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.751556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.751558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.751927 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:56.752125 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:56.752659 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 02:17:56.752870 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":901,"threads_started":1}
I20260929 02:17:56.754822 15574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:17:56 dist-test-slave-9k0h krb5kdc[15346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
Sep 29 02:17:56 dist-test-slave-9k0h krb5kdc[15346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648276, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.188@KRBTEST.COM for kudu/127.24.216.189@KRBTEST.COM
I20260929 02:17:56.767961 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.751835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.24.216.190:46373)
0929 02:17:56.752389 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:56.752393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:56.752411 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:17:56.752448 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:56.752451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:56.752497 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:56.752609 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:56.754059 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.755327 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.756804 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.757219 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.757308 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.761955 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:56.761960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:56.761963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:56.761991 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:56.765322 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.766011 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.766020 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.766023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.766172 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.766531 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.766538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.766541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.766810 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:56.767074 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:56.767390 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 02:17:56.767597 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":295,"threads_started":1}
I20260929 02:17:56.768293 15578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.755466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:39863 (local address 127.0.0.1:39242)
0929 02:17:56.755617 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.755635 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:56.755727 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:56.756066 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:56.756069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:56.756441 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:56.756567 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.756574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.757611 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.757615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.758536 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.758547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.759548 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.759552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.759692 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.760202 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:56.760218 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:56.762292 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:56.765873 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.765879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.765883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.766417 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.766780 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.766785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.766789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.766926 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.767411 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:56.767416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:56.767570 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 02:17:56.767707 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 02:17:56.767969 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 02:17:56.768518 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.755548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.24.216.189:39863)
0929 02:17:56.755732 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:56.755738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:56.755783 (+ 45us) server_negotiation.cc:407] Connection header received
0929 02:17:56.755840 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:56.755847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:56.755947 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:56.756039 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:56.756713 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.757474 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.758681 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.759399 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.759540 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.762466 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:56.762474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:56.762479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:56.762521 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:56.765748 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.766521 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.766528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.766532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.766661 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.767038 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.767043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.767046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.767295 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:56.767443 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:56.767949 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 02:17:56.768154 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:17:56.769294 15574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:17:56 dist-test-slave-9k0h krb5kdc[15346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648276, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.188@KRBTEST.COM for kudu/127.24.216.188@KRBTEST.COM
Sep 29 02:17:56 dist-test-slave-9k0h krb5kdc[15346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.189@KRBTEST.COM
I20260929 02:17:56.782840 15578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.769905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56238)
0929 02:17:56.770063 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.770081 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:56.770214 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:56.770704 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:56.770707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:56.771032 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:56.771166 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.771174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.772251 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.772256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.773306 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.773317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.774205 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.774213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.774445 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.775043 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:56.775068 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:56.777125 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:56.780275 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.780280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.780286 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.780772 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.781110 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.781115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.781118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.781321 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.781837 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:56.781842 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:56.781979 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 02:17:56.782125 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 02:17:56.782454 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 02:17:56.783145 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.770307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.24.216.188:36963)
0929 02:17:56.770432 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:56.770438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:56.770454 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:17:56.770495 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:56.770500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:56.770568 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:56.770668 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:56.771303 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.772135 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.773443 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.774015 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.774108 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.777310 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:56.777318 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:56.777326 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:56.777369 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:56.780127 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.780880 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.780884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.780886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.781001 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.781457 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.781467 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.781469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.781716 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:56.781859 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:56.782250 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 02:17:56.782670 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":33}
I20260929 02:17:56.784696 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.769773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.24.216.189:39863)
0929 02:17:56.769909 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:56.769913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:56.769927 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:17:56.770028 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:56.770033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:56.770108 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:56.770239 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:56.771690 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.772573 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.773802 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.774343 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.774461 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.778736 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:56.778740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:56.778743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:56.778770 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:56.781681 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.782422 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.782427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.782429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.782602 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.783057 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.783061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.783063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.783343 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:56.783599 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:56.784124 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 02:17:56.784372 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 02:17:56.788359 15574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf: Bootstrap starting.
I20260929 02:17:56.789291 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:17:56.791695 15574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:56.792974 15574 log.cc:824] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:56.791877 15401 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "28f1b3d9da034b40ab094d045e2dbeb3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49286
I20260929 02:17:56.795985 15574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf: No bootstrap required, opened a new log
I20260929 02:17:56.806641 15574 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } }
I20260929 02:17:56.807029 15574 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5aeb222c34c4f04bd13dcd136a15aaf, State: Initialized, Role: FOLLOWER
I20260929 02:17:56.807519 15574 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } }
I20260929 02:17:56.808360 15582 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } } }
I20260929 02:17:56.808693 15582 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:17:56.809620 15574 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:17:56.818486 15593 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:17:56.818616 15593 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:17:56.816820 15507 master_runner.cc:186] Error getting master registration for 127.24.216.190:46373: OK, instance_id { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" instance_seqno: 1790648275701301 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:17:56.824668 15507 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:17:56.838297 15426 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } attempt: 4
Sep 29 02:17:56 dist-test-slave-9k0h krb5kdc[15346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648275, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for kudu/127.24.216.188@KRBTEST.COM
I20260929 02:17:56.852526 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.838931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56244)
0929 02:17:56.839140 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.839162 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:56.839296 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:56.839637 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:56.839642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:56.840058 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:56.840226 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.840234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.841397 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.841402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.842423 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.842434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.843218 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.843224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.843433 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.844037 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:56.844054 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:56.846228 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:56.849760 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.849765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.849769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.850300 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.850703 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.850708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.850712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.850883 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.851520 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:56.851526 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:56.851704 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 02:17:56.851853 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 02:17:56.852159 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 02:17:56.852535 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.838964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.24.216.188:36963)
0929 02:17:56.839198 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:56.839206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:56.839229 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:17:56.839425 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:56.839431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:56.839507 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:56.839610 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:56.840374 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.841254 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.842569 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.843072 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.843214 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.846426 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:56.846433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:56.846443 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:56.846484 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:56.849640 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.850411 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.850417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.850421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.850576 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.850999 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.851004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.851008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.851395 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:56.851550 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:56.851975 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 02:17:56.852189 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:17:56.857450 15426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3: Bootstrap starting.
I20260929 02:17:56.860577 15426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:56.861577 15426 log.cc:824] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:56.865484 15426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3: No bootstrap required, opened a new log
I20260929 02:17:56.870594 15426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } }
I20260929 02:17:56.870955 15426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f1b3d9da034b40ab094d045e2dbeb3, State: Initialized, Role: FOLLOWER
I20260929 02:17:56.871407 15426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } }
I20260929 02:17:56.872331 15596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } } }
I20260929 02:17:56.872803 15596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:17:56.873469 15426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:56.873600 15401 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 02:17:56.873822 15401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:56.877442 15401 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } }
I20260929 02:17:56.879992 15401 leader_election.cc:288] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [CANDIDATE]: Term 1 election: Requested vote from peers df06b7847d1843ce9596ee9f8108f3d3 (127.24.216.189:39863), e5aeb222c34c4f04bd13dcd136a15aaf (127.24.216.188:36963)
I20260929 02:17:56.880035 15549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf"
I20260929 02:17:56.880555 15549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:56.884155 15549 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28f1b3d9da034b40ab094d045e2dbeb3 in term 1.
I20260929 02:17:56.884835 15377 leader_election.cc:302] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28f1b3d9da034b40ab094d045e2dbeb3, e5aeb222c34c4f04bd13dcd136a15aaf; no voters:
I20260929 02:17:56.885317 15596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:17:56.896620 15596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [term 1 LEADER]: Becoming Leader. State: Replica: 28f1b3d9da034b40ab094d045e2dbeb3, State: Running, Role: LEADER
I20260929 02:17:56.897290 15596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } }
I20260929 02:17:56.906328 15607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28f1b3d9da034b40ab094d045e2dbeb3. Latest consensus state: current_term: 1 leader_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } } }
I20260929 02:17:56.907943 15607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:56.922653 15500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } attempt: 5
W20260929 02:17:56.940824 15380 tablet.cc:2369] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:17:56.944566 15615 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:17:56.944694 15615 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:17:56.944821 15616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 02:17:56 dist-test-slave-9k0h krb5kdc[15346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648276, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.189@KRBTEST.COM for kudu/127.24.216.188@KRBTEST.COM
I20260929 02:17:56.948094 15616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:17:56.924830 15359 master_runner.cc:186] Error getting master registration for 127.24.216.189:39863: OK, instance_id { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" instance_seqno: 1790648276196369 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:17:56.949491 15359 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 02:17:56.954562 15528 tablet.cc:2369] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:17:56.960213 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.923176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:36963 (local address 127.0.0.1:56256)
0929 02:17:56.923524 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 02:17:56.923542 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:17:56.923645 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:17:56.938985 (+ 15340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:17:56.938992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:17:56.939501 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:17:56.939707 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.939720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.941699 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.941706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.943083 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:17:56.943096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.944053 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.944058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:17:56.944320 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.945001 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:17:56.945022 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:17:56.947912 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:17:56.957227 (+ 9315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.957232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.957235 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.957722 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.958172 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:17:56.958188 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:17:56.958193 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:17:56.958362 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:17:56.959069 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:17:56.959075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:17:56.959242 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 02:17:56.959402 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 02:17:56.959693 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260929 02:17:56.961417 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.923855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.24.216.188:36963)
0929 02:17:56.938603 (+ 14748us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:56.938609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:56.938634 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:17:56.938696 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:56.938702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:56.938788 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:56.939927 (+ 1139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:56.940218 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.941528 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.943286 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.943885 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.948088 (+ 4203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.948339 (+ 251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:56.948345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:56.948352 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:56.948387 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:56.957074 (+ 8687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.957874 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.957878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.957881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.958027 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.958512 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.958517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.958520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.958917 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:56.960589 (+ 1672us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:56.960718 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 02:17:56.960925 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14604,"thread_start_us":15701,"threads_started":1}
I20260929 02:17:56.967970 15500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3: Bootstrap starting.
I20260929 02:17:56.973055 15500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3: Neither blocks nor log segments found. Creating new log.
I20260929 02:17:56.978122 15500 log.cc:824] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:17:56.982049 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.906850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.24.216.190:46373)
0929 02:17:56.907039 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:56.907046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:56.918812 (+ 11766us) server_negotiation.cc:407] Connection header received
0929 02:17:56.936999 (+ 18187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:56.937005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:56.937097 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:56.937174 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:56.945746 (+ 8572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.947012 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.952525 (+ 5513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.953402 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.953527 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.962092 (+ 8565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:56.962103 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:56.962110 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:56.962187 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:56.975380 (+ 13193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.977252 (+ 1872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.977262 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.977269 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.977453 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.978418 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.978427 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.978433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.978899 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:56.979214 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:56.980557 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0929 02:17:56.981101 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 02:17:56.987423 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.906567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.24.216.189:39863)
0929 02:17:56.906770 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:56.906779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:56.936511 (+ 29732us) server_negotiation.cc:407] Connection header received
0929 02:17:56.939952 (+ 3441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:56.939957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:56.940052 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:56.940123 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:56.950600 (+ 10477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.951706 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.955505 (+ 3799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.956175 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.956269 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.961549 (+ 5280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:56.961557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:56.961561 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:56.961599 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:56.974083 (+ 12484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.984163 (+ 10080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.984169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.984173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.984322 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.984747 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.984752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.984756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.985089 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:56.985329 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:56.986097 (+ 768us) server_negotiation.cc:299] Negotiation successful
0929 02:17:56.986658 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:17:56.988610 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:56.921612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.24.216.188:36963)
0929 02:17:56.921805 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:56.921812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:56.946816 (+ 25004us) server_negotiation.cc:407] Connection header received
0929 02:17:56.947043 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:56.947049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:56.947118 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:56.947183 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:56.949288 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.950462 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.957223 (+ 6761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:56.957870 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:56.957960 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:56.979261 (+ 21301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:17:56.979268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:17:56.979273 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:17:56.979315 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:17:56.983169 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.986209 (+ 3040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.986214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.986217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.986388 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:17:56.986861 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:17:56.986866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:17:56.986870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:17:56.987195 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:17:56.987418 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:56.988032 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 02:17:56.988271 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:17:56.995263 15500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3: No bootstrap required, opened a new log
I20260929 02:17:56.995468 15549 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf [term 1 FOLLOWER]: Refusing update from remote peer 28f1b3d9da034b40ab094d045e2dbeb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:17:56.996429 15607 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:17:57.012022 15500 raft_consensus.cc:348] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } }
I20260929 02:17:57.012775 15500 raft_consensus.cc:729] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df06b7847d1843ce9596ee9f8108f3d3, State: Initialized, Role: FOLLOWER
I20260929 02:17:57.013653 15500 consensus_queue.cc:260] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } }
I20260929 02:17:57.017083 15582 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28f1b3d9da034b40ab094d045e2dbeb3. Latest consensus state: current_term: 1 leader_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } } }
I20260929 02:17:57.017426 15582 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:17:57.019557 15623 sys_catalog.cc:455] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } } }
I20260929 02:17:57.020278 15623 sys_catalog.cc:458] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:17:57.021019 15500 sys_catalog.cc:564] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:17:57.021054 15475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "df06b7847d1843ce9596ee9f8108f3d3"
I20260929 02:17:57.021504 15475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:17:57.025254 15475 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28f1b3d9da034b40ab094d045e2dbeb3 in term 1.
I20260929 02:17:57.026077 15474 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3 [term 1 FOLLOWER]: Refusing update from remote peer 28f1b3d9da034b40ab094d045e2dbeb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:17:57.028180 15607 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:17:57.038671 15622 mvcc.cc:205] Tried to move back new op lower bound from 7334495342559997952 to 7334495342177337344. Current Snapshot: MvccSnapshot[applied={T|T < 7334495342559997952}]
I20260929 02:17:57.039325 15582 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } } }
I20260929 02:17:57.039569 15582 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:17:57.048305 15607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } } }
I20260929 02:17:57.048662 15607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:57.050447 15616 catalog_manager.cc:1347] Generated new cluster ID: 8e3c718bb12c4c97bfeaf2dba62a35c7
I20260929 02:17:57.050570 15616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:17:57.070237 15623 sys_catalog.cc:455] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28f1b3d9da034b40ab094d045e2dbeb3. Latest consensus state: current_term: 1 leader_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } } }
I20260929 02:17:57.070596 15623 sys_catalog.cc:458] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:17:57.074647 15616 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:17:57.075999 15616 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:17:57.084233 15596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } } }
I20260929 02:17:57.084631 15596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:17:57.086704 15628 mvcc.cc:205] Tried to move back new op lower bound from 7334495342559997952 to 7334495342177337344. Current Snapshot: MvccSnapshot[applied={T|T < 7334495342559997952}]
I20260929 02:17:57.091470 15623 sys_catalog.cc:455] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f1b3d9da034b40ab094d045e2dbeb3" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 46373 } } peers { permanent_uuid: "df06b7847d1843ce9596ee9f8108f3d3" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 39863 } } peers { permanent_uuid: "e5aeb222c34c4f04bd13dcd136a15aaf" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 36963 } } }
I20260929 02:17:57.091790 15623 sys_catalog.cc:458] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:17:57.094287 15616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28f1b3d9da034b40ab094d045e2dbeb3: Generated new TSK 0
I20260929 02:17:57.095321 15616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:17:57.149390 15641 catalog_manager.cc:1259] Loaded cluster ID: 8e3c718bb12c4c97bfeaf2dba62a35c7
I20260929 02:17:57.149590 15641 catalog_manager.cc:1552] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3: loading cluster ID for follower catalog manager: success
I20260929 02:17:57.153939 15641 catalog_manager.cc:1574] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3: acquiring CA information for follower catalog manager: success
I20260929 02:17:57.156381 15641 catalog_manager.cc:1602] T 00000000000000000000000000000000 P df06b7847d1843ce9596ee9f8108f3d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 02:17:57.258185 15580 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:17:57.240349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.24.216.190:46373)
0929 02:17:57.240546 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.240552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.252245 (+ 11693us) server_negotiation.cc:407] Connection header received
0929 02:17:57.255486 (+ 3241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.255496 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.255619 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.255720 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:57.257765 (+ 2045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49302: BlockingRecv error: recv got EOF from 127.0.0.1:49302 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 02:17:57.261638 15504 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:17:57.246054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.24.216.189:39863)
0929 02:17:57.246336 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.246346 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.259594 (+ 13248us) server_negotiation.cc:407] Connection header received
0929 02:17:57.260229 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.260239 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.260324 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.260393 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:57.261418 (+ 1025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39256: BlockingRecv error: recv got EOF from 127.0.0.1:39256 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 02:17:57.262135 15577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:17:57.252439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.24.216.188:36963)
0929 02:17:57.252654 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.252662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.259595 (+ 6933us) server_negotiation.cc:407] Connection header received
0929 02:17:57.260319 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.260325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.260407 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.260479 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:57.261418 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56266: BlockingRecv error: recv got EOF from 127.0.0.1:56266 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:17:57.317649 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:57.295499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39266 (local address 127.24.216.189:39863)
0929 02:17:57.295712 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.295719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.309903 (+ 14184us) server_negotiation.cc:407] Connection header received
0929 02:17:57.310218 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.310224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.310425 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.310592 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:57.311342 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.312491 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.314566 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.315272 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.315375 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:57.316145 (+ 770us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:57.316824 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:57.316938 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:57.317181 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 02:17:57.317268 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 02:17:57.360926 15577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:17:57.350718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.24.216.188:36963)
0929 02:17:57.350924 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.350937 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.358763 (+ 7826us) server_negotiation.cc:407] Connection header received
0929 02:17:57.359305 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.359311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.359466 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.359555 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:57.360663 (+ 1108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56282: BlockingRecv error: recv got EOF from 127.0.0.1:56282 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:17:57.407881 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:57.393673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.24.216.189:39863)
0929 02:17:57.393926 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.393933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.400749 (+ 6816us) server_negotiation.cc:407] Connection header received
0929 02:17:57.401281 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.401287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.401387 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.401461 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:57.402227 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.403392 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.405391 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.406089 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.406237 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:57.406866 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:57.407065 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:57.407138 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:57.407393 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 02:17:57.407497 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 02:17:57.451668 15577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:17:57.441951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.24.216.188:36963)
0929 02:17:57.442193 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.442199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.449527 (+ 7328us) server_negotiation.cc:407] Connection header received
0929 02:17:57.450037 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.450046 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.450187 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.450273 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:57.451387 (+ 1114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56294: BlockingRecv error: recv got EOF from 127.0.0.1:56294 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:17:57.498098 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:57.484008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.24.216.189:39863)
0929 02:17:57.484203 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.484210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.491102 (+ 6892us) server_negotiation.cc:407] Connection header received
0929 02:17:57.491638 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.491645 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.491763 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.491837 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:57.492543 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.493729 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.495793 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.496464 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.496559 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:57.497166 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:57.497356 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:57.497428 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:57.497661 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 02:17:57.497764 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 02:17:57.542184 15577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:17:57.532647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56306 (local address 127.24.216.188:36963)
0929 02:17:57.532850 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.532857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.540192 (+ 7335us) server_negotiation.cc:407] Connection header received
0929 02:17:57.540671 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.540689 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.540786 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.540861 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:57.541914 (+ 1053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56306: BlockingRecv error: recv got EOF from 127.0.0.1:56306 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:17:57.589958 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:57.583284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39304 (local address 127.24.216.189:39863)
0929 02:17:57.583535 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.583542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.584091 (+ 549us) server_negotiation.cc:407] Connection header received
0929 02:17:57.584248 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.584254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.584365 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.584475 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:57.584950 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.586126 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.587998 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.588725 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.588870 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:57.589229 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:57.589373 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:57.589471 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:57.589581 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 02:17:57.589676 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
W20260929 02:17:57.600769 15577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:17:57.598598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.24.216.188:36963)
0929 02:17:57.598804 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.598812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.599293 (+ 481us) server_negotiation.cc:407] Connection header received
0929 02:17:57.599504 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.599512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.599602 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.599710 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:57.600524 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56312: BlockingRecv error: recv got EOF from 127.0.0.1:56312 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 02:17:57.617995 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:57.612252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.24.216.189:39863)
0929 02:17:57.612466 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.612472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.612811 (+ 339us) server_negotiation.cc:407] Connection header received
0929 02:17:57.612990 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.612996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.613091 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.613198 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:57.613568 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.614727 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.616021 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.616657 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.616795 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:57.617162 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:57.617322 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:57.617424 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:57.617553 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 02:17:57.617659 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260929 02:17:57.625643 15577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:17:57.623604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.24.216.188:36963)
0929 02:17:57.623768 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.623772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.624243 (+ 471us) server_negotiation.cc:407] Connection header received
0929 02:17:57.624462 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.624466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.624540 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.624644 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:57.625428 (+ 784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56326: BlockingRecv error: recv got EOF from 127.0.0.1:56326 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:17:57.668843 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:57.663007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.24.216.189:39863)
0929 02:17:57.663201 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.663210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.663545 (+ 335us) server_negotiation.cc:407] Connection header received
0929 02:17:57.663707 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.663712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.663806 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.663929 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:57.664299 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.665558 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.666902 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.667575 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.667730 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:57.668079 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:57.668216 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:57.668306 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:57.668441 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 02:17:57.668521 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 02:17:57.677103 15577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:17:57.675286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.24.216.188:36963)
0929 02:17:57.675479 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.675485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.675796 (+ 311us) server_negotiation.cc:407] Connection header received
0929 02:17:57.675942 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.675947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.676049 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.676159 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:57.676866 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56340: BlockingRecv error: recv got EOF from 127.0.0.1:56340 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:17:57.718724 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:57.713161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.24.216.189:39863)
0929 02:17:57.713350 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.713357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.713899 (+ 542us) server_negotiation.cc:407] Connection header received
0929 02:17:57.714126 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.714132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.714288 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.714389 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:57.714807 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.715536 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.716853 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.717370 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.717508 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:57.717897 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:57.717995 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:57.718094 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:57.718301 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 02:17:57.718377 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 02:17:57.736065 15577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:17:57.726163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.24.216.188:36963)
0929 02:17:57.726366 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.726373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.734777 (+ 8404us) server_negotiation.cc:407] Connection header received
0929 02:17:57.734903 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.734908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.735014 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.735116 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:57.735851 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56344: BlockingRecv error: recv got EOF from 127.0.0.1:56344 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:17:57.808852 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:57.803690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.24.216.189:39863)
0929 02:17:57.803935 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.803942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.804199 (+ 257us) server_negotiation.cc:407] Connection header received
0929 02:17:57.804444 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.804449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.804515 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.804619 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:57.805046 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.805799 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.807169 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.807629 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.807772 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:57.808156 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:57.808264 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:57.808365 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:57.808516 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 02:17:57.808593 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 02:17:57.816665 15577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:17:57.814957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56348 (local address 127.24.216.188:36963)
0929 02:17:57.815223 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.815229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.815475 (+ 246us) server_negotiation.cc:407] Connection header received
0929 02:17:57.815629 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.815633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.815706 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.815786 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:17:57.816460 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56348: BlockingRecv error: recv got EOF from 127.0.0.1:56348 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 02:17:57.820449 15593 catalog_manager.cc:1259] Loaded cluster ID: 8e3c718bb12c4c97bfeaf2dba62a35c7
I20260929 02:17:57.820529 15593 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf: loading cluster ID for follower catalog manager: success
I20260929 02:17:57.823028 15593 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf: acquiring CA information for follower catalog manager: success
I20260929 02:17:57.824761 15593 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5aeb222c34c4f04bd13dcd136a15aaf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:17:57.957083 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:57.951822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.24.216.189:39863)
0929 02:17:57.951988 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.951992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.952340 (+ 348us) server_negotiation.cc:407] Connection header received
0929 02:17:57.952507 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.952511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.952579 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.952668 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:57.953069 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.953767 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.955184 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.955666 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.955788 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:57.956266 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:57.956426 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:57.956493 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:57.956694 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 02:17:57.956783 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:17:57.968309 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:57.962800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56362 (local address 127.24.216.188:36963)
0929 02:17:57.963026 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.963032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.963281 (+ 249us) server_negotiation.cc:407] Connection header received
0929 02:17:57.963486 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.963491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.963667 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.963797 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:57.964185 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.964871 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.966194 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.966595 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.966751 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:57.967230 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:57.967742 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:57.967903 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:57.967963 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 02:17:57.968040 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:17:57.982218 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:57.974227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.24.216.189:39863)
0929 02:17:57.974484 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.974494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.975700 (+ 1206us) server_negotiation.cc:407] Connection header received
0929 02:17:57.976031 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.976037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.976126 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.976204 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:57.977343 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.978410 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.980024 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.980536 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.980653 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:57.981226 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:57.981360 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:57.981467 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:57.981768 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 02:17:57.981858 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:17:57.983019 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:57.974017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49312 (local address 127.24.216.190:46373)
0929 02:17:57.974964 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.974971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.975420 (+ 449us) server_negotiation.cc:407] Connection header received
0929 02:17:57.975802 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.975809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.976090 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.976277 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:57.976988 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.978129 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.980475 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.981043 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.981196 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:57.981865 (+ 669us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:57.982357 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:57.982472 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:57.982655 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 02:17:57.982738 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":653,"threads_started":1}
I20260929 02:17:57.984618 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:17:57.975850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.24.216.188:36963)
0929 02:17:57.976080 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:17:57.976088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:17:57.977147 (+ 1059us) server_negotiation.cc:407] Connection header received
0929 02:17:57.977445 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:17:57.977451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:17:57.977543 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:17:57.977614 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:17:57.978520 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.979701 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.982122 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:17:57.982872 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:17:57.982988 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:17:57.983807 (+ 819us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:17:57.983933 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:17:57.984036 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 02:17:57.984305 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 02:17:57.984387 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 02:17:57.988912 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 15359
I20260929 02:17:57.989889 15421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:58.101488 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 15359
I20260929 02:17:58.122972 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 15433
I20260929 02:17:58.124089 15495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:58.230387 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 15433
I20260929 02:17:58.250433 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 15507
I20260929 02:17:58.251538 15569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:17:58.359937 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 15507
2026-09-29T02:17:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4480 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:17:58 dist-test-slave-9k0h krb5kdc[15754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:17:58 dist-test-slave-9k0h krb5kdc[15754](info): set up 2 sockets
Sep 29 02:17:58 dist-test-slave-9k0h krb5kdc[15754](info): commencing operation
krb5kdc: starting...
W20260929 02:17:59.478672 25442 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/test-user.keytab.
Sep 29 02:17:59 dist-test-slave-9k0h krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:17:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:17:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.216.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
Entry for principal kudu/127.24.216.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab.
I20260929 02:17:59.757529 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:41223
--webserver_interface=127.24.216.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:41223,127.24.216.189:43923,127.24.216.188:38227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:18:00.082325 15767 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:18:00.082656 15767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:18:00.082775 15767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:18:00.082901 15767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:18:00.089977 15767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:18:00.090065 15767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:18:00.090185 15767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:18:00.090272 15767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:18:00.090399 15767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:18:00.100253 15767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:41223,127.24.216.189:43923,127.24.216.188:38227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.190.keytab
--principal=kudu/127.24.216.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.190:41223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.216.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:18:00.101542 15767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:18:00.102970 15767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:18:00.114560 15773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:18:00.114634 15775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:18:00.114668 15772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:18:00.115661 15767 server_base.cc:1034] running on GCE node
I20260929 02:18:00.116398 15767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:18:00.117699 15767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:18:00.118934 15767 hybrid_clock.cc:648] HybridClock initialized: now 1790648280118884 us; error 48 us; skew 500 ppm
I20260929 02:18:00.119280 15767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:18:00 dist-test-slave-9k0h krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648280, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:18:00.122437 15767 init.cc:377] Logged in from keytab as kudu/127.24.216.190@KRBTEST.COM (short username kudu)
I20260929 02:18:00.123874 15767 webserver.cc:460] Webserver started at http://127.24.216.190:46127/ using document root <none> and password file <none>
I20260929 02:18:00.124552 15767 fs_manager.cc:359] Metadata directory not provided
I20260929 02:18:00.124686 15767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:18:00.124997 15767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:18:00.127892 15767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/data/instance:
uuid: "eb6d3ff2e62043c8a9671be3aa27d819"
format_stamp: "Formatted at 2026-09-29 02:18:00 on dist-test-slave-9k0h"
server_key: "d6054fe101d58217d5e34ac2cb30089b"
server_key_iv: "9cacd502287f0b6e1beb306d39652797"
server_key_version: "encryptionkey@0"
I20260929 02:18:00.128691 15767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/wal/instance:
uuid: "eb6d3ff2e62043c8a9671be3aa27d819"
format_stamp: "Formatted at 2026-09-29 02:18:00 on dist-test-slave-9k0h"
server_key: "d6054fe101d58217d5e34ac2cb30089b"
server_key_iv: "9cacd502287f0b6e1beb306d39652797"
server_key_version: "encryptionkey@0"
I20260929 02:18:00.133633 15767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:18:00.136850 15782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:18:00.138064 15767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 02:18:00.138289 15767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/wal
uuid: "eb6d3ff2e62043c8a9671be3aa27d819"
format_stamp: "Formatted at 2026-09-29 02:18:00 on dist-test-slave-9k0h"
server_key: "d6054fe101d58217d5e34ac2cb30089b"
server_key_iv: "9cacd502287f0b6e1beb306d39652797"
server_key_version: "encryptionkey@0"
I20260929 02:18:00.138512 15767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:18:00.167546 15767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:18:00.171998 15767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:18:00.172287 15767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:18:00.190786 15767 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.190:41223
I20260929 02:18:00.190806 15833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.190:41223 every 8 connection(s)
I20260929 02:18:00.192095 15767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/data/info.pb
I20260929 02:18:00.194219 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 15767
I20260929 02:18:00.194411 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-0/wal/instance
I20260929 02:18:00.194988 25442 external_mini_cluster.cc:1428] Setting key fc2f65cb2bffa83dffc960e8e11a22b1
I20260929 02:18:00.196362 15834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:18:00.206396 15834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:18:00 dist-test-slave-9k0h krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648280, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.216.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.189@KRBTEST.COM" created.
I20260929 02:18:00.230947 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.210034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.24.216.190:41223)
0929 02:18:00.211722 (+ 1688us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:00.211731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:00.211784 (+ 53us) server_negotiation.cc:407] Connection header received
0929 02:18:00.213340 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:00.213350 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:00.213636 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:00.213836 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:00.215112 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.217039 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.218495 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.219413 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.219604 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:00.223294 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:00.223308 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:00.223328 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:00.223373 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:00.226928 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:00.227568 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:00.227574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:00.227576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:00.227712 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:00.228128 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:00.228135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:00.228138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:00.228730 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:00.228928 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:00.229556 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 02:18:00.229825 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1474,"spinlock_wait_cycles":13184,"thread_start_us":1234,"threads_started":1}
I20260929 02:18:00.230983 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.210034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:41223 (local address 127.0.0.1:52084)
0929 02:18:00.211049 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.211120 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 02:18:00.212981 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:18:00.213866 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:18:00.213871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:18:00.214556 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:18:00.214884 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:00.214900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.217200 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.217204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:00.218356 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:00.218367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.219667 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.219673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:00.219819 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:00.220538 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:18:00.220561 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:18:00.223025 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:18:00.227051 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:00.227061 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:00.227072 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:00.227443 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:00.227807 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:00.227810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:00.227812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:00.227955 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:00.228859 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:18:00.228866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:18:00.229113 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 02:18:00.229397 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 02:18:00.229993 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":81,"thread_start_us":403,"threads_started":1}
I20260929 02:18:00.233927 15834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:18:00.235064 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.234608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:43923 (local address 127.0.0.1:59014)
0929 02:18:00.234800 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.234925 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 02:18:00.235347 15786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
I20260929 02:18:00.236924 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.236506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:43923 (local address 127.0.0.1:59016)
0929 02:18:00.236678 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.236765 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 02:18:00.237166 15834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:43923: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.189.keytab.
Entry for principal kudu/127.24.216.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.189.keytab.
I20260929 02:18:00.264508 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.189:43923
--webserver_interface=127.24.216.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.189.keytab
--principal=kudu/127.24.216.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:41223,127.24.216.189:43923,127.24.216.188:38227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:18:00.286552 15834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } attempt: 1
I20260929 02:18:00.287894 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.287420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:43923 (local address 127.0.0.1:59030)
0929 02:18:00.287631 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.287761 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 02:18:00.288872 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.288524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:43923 (local address 127.0.0.1:59032)
0929 02:18:00.288688 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.288754 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 02:18:00.289058 15834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:43923: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
I20260929 02:18:00.357322 15834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } attempt: 2
I20260929 02:18:00.358537 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.358018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:43923 (local address 127.0.0.1:59034)
0929 02:18:00.358315 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.358401 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 02:18:00.359511 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.359162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:43923 (local address 127.0.0.1:59046)
0929 02:18:00.359302 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.359385 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 02:18:00.359684 15834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:43923: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
I20260929 02:18:00.450982 15834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } attempt: 3
I20260929 02:18:00.452106 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.451664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:43923 (local address 127.0.0.1:59060)
0929 02:18:00.451860 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.451956 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 02:18:00.453047 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.452711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:43923 (local address 127.0.0.1:59076)
0929 02:18:00.452854 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.452924 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:18:00.453227 15834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:43923: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
W20260929 02:18:00.594488 15841 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:18:00.594843 15841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:18:00.594965 15841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:18:00.595104 15841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 02:18:00.596462 15834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } attempt: 4
I20260929 02:18:00.597630 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.597120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:43923 (local address 127.0.0.1:59080)
0929 02:18:00.597420 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.597501 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260929 02:18:00.598596 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.598276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:43923 (local address 127.0.0.1:59082)
0929 02:18:00.598409 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.598491 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 02:18:00.598768 15834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.189:43923: Network error: Client connection negotiation failed: client connection to 127.24.216.189:43923: connect: Connection refused (error 111)
W20260929 02:18:00.603109 15841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:18:00.603185 15841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:18:00.603241 15841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:18:00.603293 15841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:18:00.603360 15841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:18:00.613054 15841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:41223,127.24.216.189:43923,127.24.216.188:38227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.189.keytab
--principal=kudu/127.24.216.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.189:43923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.216.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:18:00.614318 15841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:18:00.615553 15841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:18:00.627206 15846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:18:00.627134 15847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:18:00.627170 15849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:18:00.627774 15841 server_base.cc:1034] running on GCE node
I20260929 02:18:00.628523 15841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:18:00.629869 15841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:18:00.631111 15841 hybrid_clock.cc:648] HybridClock initialized: now 1790648280631036 us; error 75 us; skew 500 ppm
I20260929 02:18:00.631428 15841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:18:00 dist-test-slave-9k0h krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648280, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:18:00.634433 15841 init.cc:377] Logged in from keytab as kudu/127.24.216.189@KRBTEST.COM (short username kudu)
I20260929 02:18:00.635850 15841 webserver.cc:460] Webserver started at http://127.24.216.189:38021/ using document root <none> and password file <none>
I20260929 02:18:00.636648 15841 fs_manager.cc:359] Metadata directory not provided
I20260929 02:18:00.636777 15841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:18:00.637094 15841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:18:00.640025 15841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/data/instance:
uuid: "6b2f1e5bdb9a43a7acaa454582727a17"
format_stamp: "Formatted at 2026-09-29 02:18:00 on dist-test-slave-9k0h"
server_key: "4abc2f5a77580c0875729f3ac3957bb5"
server_key_iv: "37e8eaf49cad841a003f4faa2bb68b47"
server_key_version: "encryptionkey@0"
I20260929 02:18:00.640805 15841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/wal/instance:
uuid: "6b2f1e5bdb9a43a7acaa454582727a17"
format_stamp: "Formatted at 2026-09-29 02:18:00 on dist-test-slave-9k0h"
server_key: "4abc2f5a77580c0875729f3ac3957bb5"
server_key_iv: "37e8eaf49cad841a003f4faa2bb68b47"
server_key_version: "encryptionkey@0"
I20260929 02:18:00.646852 15841 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 02:18:00.650584 15856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:18:00.651986 15841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:18:00.652149 15841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/wal
uuid: "6b2f1e5bdb9a43a7acaa454582727a17"
format_stamp: "Formatted at 2026-09-29 02:18:00 on dist-test-slave-9k0h"
server_key: "4abc2f5a77580c0875729f3ac3957bb5"
server_key_iv: "37e8eaf49cad841a003f4faa2bb68b47"
server_key_version: "encryptionkey@0"
I20260929 02:18:00.652357 15841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:18:00.672855 15841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:18:00.677125 15841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:18:00.677418 15841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:18:00.696489 15841 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.189:43923
I20260929 02:18:00.696506 15907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.189:43923 every 8 connection(s)
I20260929 02:18:00.697904 15841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/data/info.pb
I20260929 02:18:00.701445 15908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:18:00.702140 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 15841
I20260929 02:18:00.702381 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-1/wal/instance
I20260929 02:18:00.702808 25442 external_mini_cluster.cc:1428] Setting key 609605705d7226225f58b510e9bf519f
I20260929 02:18:00.707871 15908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:18:00 dist-test-slave-9k0h krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648280, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.189@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:18:00.727895 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.710536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.24.216.190:41223)
0929 02:18:00.710777 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:00.710785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:00.711896 (+ 1111us) server_negotiation.cc:407] Connection header received
0929 02:18:00.713688 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:00.713694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:00.713779 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:00.713887 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:00.715371 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.716068 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.717510 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.717911 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.718035 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:00.721940 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:00.721945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:00.721948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:00.721979 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:00.724583 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:00.725544 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:00.725547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:00.725550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:00.725666 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:00.726188 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:00.726199 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:00.726201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:00.726484 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:00.726600 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:00.727501 (+ 901us) server_negotiation.cc:299] Negotiation successful
0929 02:18:00.727673 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
WARNING: no policy specified for kudu/127.24.216.188@KRBTEST.COM; defaulting to no policy
I20260929 02:18:00.729341 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.710967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:41223 (local address 127.0.0.1:52100)
0929 02:18:00.711714 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.711764 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:18:00.713422 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:18:00.714130 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:18:00.714138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:18:00.714702 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:18:00.715197 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:00.715213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.716221 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.716234 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:00.717372 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:00.717382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.718174 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.718183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:00.718381 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:00.719051 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:18:00.719083 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:18:00.721722 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:18:00.724754 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:00.724765 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:00.724782 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:00.725425 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:00.725772 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:00.725779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:00.725784 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:00.726030 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:00.726611 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:18:00.726622 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:18:00.727095 (+ 473us) client_negotiation.cc:769] Sending connection context
0929 02:18:00.727413 (+ 318us) client_negotiation.cc:240] Negotiation successful
0929 02:18:00.727937 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":320,"threads_started":1}
I20260929 02:18:00.731053 15908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.216.188@KRBTEST.COM" created.
Sep 29 02:18:00 dist-test-slave-9k0h krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648280, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.189@KRBTEST.COM for kudu/127.24.216.189@KRBTEST.COM
I20260929 02:18:00.746944 15912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.731959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59088 (local address 127.24.216.189:43923)
0929 02:18:00.732704 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:00.732713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:00.732741 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:18:00.732789 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:00.732796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:00.733024 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:00.733168 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:00.733845 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.735641 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.736722 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.737603 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.737736 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:00.740802 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:00.740816 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:00.740834 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:00.740874 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:00.744139 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:00.744915 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:00.744921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:00.744923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:00.745024 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:00.745434 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:00.745437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:00.745439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:00.745738 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:00.745886 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:00.746365 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 02:18:00.746639 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":581,"thread_start_us":439,"threads_started":1}
I20260929 02:18:00.747102 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.731858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:43923 (local address 127.0.0.1:59088)
0929 02:18:00.732075 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.732095 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:18:00.732179 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:18:00.733174 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:18:00.733177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:18:00.733530 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:18:00.733668 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:00.733675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.735770 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.735773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:00.736579 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:00.736588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.737742 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.737745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:00.737858 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:00.738466 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:18:00.738483 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:18:00.740574 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:18:00.744272 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:00.744279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:00.744286 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:00.744795 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:00.745154 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:00.745161 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:00.745168 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:00.745332 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:00.745859 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:18:00.745867 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:18:00.746026 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 02:18:00.746208 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 02:18:00.746528 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 02:18:00.749580 15908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:18:00.750767 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.750373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35194)
0929 02:18:00.750518 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.750622 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:18:00.751013 15857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
I20260929 02:18:00.752534 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.752221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35206)
0929 02:18:00.752384 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.752449 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:18:00.752739 15908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:38227: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.216.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.188.keytab.
Entry for principal kudu/127.24.216.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.188.keytab.
I20260929 02:18:00.769593 25442 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
/tmp/dist-test-taskGNPMkl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.188:38227
--webserver_interface=127.24.216.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.188.keytab
--principal=kudu/127.24.216.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.216.148:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.216.190:41223,127.24.216.189:43923,127.24.216.188:38227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:18:00.802008 15908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } attempt: 1
I20260929 02:18:00.803251 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.802822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35220)
0929 02:18:00.803011 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.803110 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 02:18:00.804296 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.803992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35224)
0929 02:18:00.804133 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.804201 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 02:18:00.804556 15908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:38227: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
I20260929 02:18:00.872762 15908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } attempt: 2
I20260929 02:18:00.873853 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.873409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35236)
0929 02:18:00.873605 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.873694 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 02:18:00.874816 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.874481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35242)
0929 02:18:00.874625 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.874706 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 02:18:00.875002 15908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:38227: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
I20260929 02:18:00.898010 15834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } attempt: 5
Sep 29 02:18:00 dist-test-slave-9k0h krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648280, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for kudu/127.24.216.189@KRBTEST.COM
I20260929 02:18:00.912206 15912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.898934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.24.216.189:43923)
0929 02:18:00.899129 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:00.899137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:00.899160 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:18:00.899369 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:00.899381 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:00.899461 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:00.899571 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:00.900384 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.901215 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.902486 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.902933 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.903070 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:00.906471 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:00.906476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:00.906479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:00.906510 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:00.909299 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:00.910224 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:00.910228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:00.910230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:00.910372 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:00.910779 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:00.910782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:00.910784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:00.911054 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:00.911237 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:00.911644 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 02:18:00.911863 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:18:00.912221 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.898821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:43923 (local address 127.0.0.1:59092)
0929 02:18:00.899049 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.899071 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:18:00.899226 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:18:00.899606 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:18:00.899610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:18:00.900030 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:18:00.900222 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:00.900231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.901343 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.901350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:00.902340 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:00.902352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:00.903083 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:00.903089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:00.903316 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:00.903953 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:18:00.903971 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:18:00.906246 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:18:00.909467 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:00.909472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:00.909477 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:00.910064 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:00.910483 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:00.910489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:00.910494 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:00.910676 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:00.911192 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:18:00.911196 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:18:00.911376 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 02:18:00.911538 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 02:18:00.911821 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 02:18:00.913380 15834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:18:00.914521 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.913986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35250)
0929 02:18:00.914212 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.914324 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 02:18:00.915509 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.915138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35264)
0929 02:18:00.915301 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.915391 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 02:18:00.915725 15834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:38227: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
I20260929 02:18:00.966213 15908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } attempt: 3
I20260929 02:18:00.966946 15834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } attempt: 1
I20260929 02:18:00.967595 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.967012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35268)
0929 02:18:00.967279 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.967399 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 02:18:00.968043 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.967473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35278)
0929 02:18:00.967722 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.967836 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 02:18:00.968787 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.968295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35284)
0929 02:18:00.968504 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.968617 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260929 02:18:00.969079 15908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:38227: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
I20260929 02:18:00.969170 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:00.968734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35298)
0929 02:18:00.968918 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:00.969024 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 02:18:00.969380 15834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:38227: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
I20260929 02:18:01.037608 15834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } attempt: 2
I20260929 02:18:01.038888 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.038303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35314)
0929 02:18:01.038558 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:01.038677 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 02:18:01.039985 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.039569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35320)
0929 02:18:01.039756 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:01.039845 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 02:18:01.040210 15834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:38227: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
I20260929 02:18:01.112330 15908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } attempt: 4
I20260929 02:18:01.113459 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.112981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35322)
0929 02:18:01.113221 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:01.113305 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 02:18:01.114451 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.114109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35332)
0929 02:18:01.114288 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:01.114352 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260929 02:18:01.114683 15908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:38227: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
W20260929 02:18:01.114951 15915 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:18:01.115281 15915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:18:01.115384 15915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:18:01.115525 15915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:18:01.123052 15915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:18:01.123139 15915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:18:01.123247 15915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:18:01.123306 15915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:18:01.123399 15915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:18:01.133518 15915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.216.148:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.216.190:41223,127.24.216.189:43923,127.24.216.188:38227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/krb5kdc/kudu_127.24.216.188.keytab
--principal=kudu/127.24.216.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.216.188:38227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.216.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:05:47 UTC on ccycloud.quasar-uepxvq.root.comops.site
build id 4679060
ASAN enabled
I20260929 02:18:01.134877 15915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:18:01.136212 15915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 02:18:01.146437 15834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } attempt: 3
I20260929 02:18:01.147809 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.147131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35346)
0929 02:18:01.147486 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:01.147597 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
W20260929 02:18:01.148535 15920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:18:01.149106 15923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:18:01.149160 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.148605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35358)
0929 02:18:01.148870 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:01.149013 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260929 02:18:01.149135 15921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:18:01.149382 15834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.216.188:38227: Network error: Client connection negotiation failed: client connection to 127.24.216.188:38227: connect: Connection refused (error 111)
I20260929 02:18:01.150493 15915 server_base.cc:1034] running on GCE node
I20260929 02:18:01.151332 15915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:18:01.152807 15915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:18:01.154085 15915 hybrid_clock.cc:648] HybridClock initialized: now 1790648281153966 us; error 62 us; skew 500 ppm
I20260929 02:18:01.154460 15915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:18:01 dist-test-slave-9k0h krb5kdc[15754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648281, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:18:01.157795 15915 init.cc:377] Logged in from keytab as kudu/127.24.216.188@KRBTEST.COM (short username kudu)
I20260929 02:18:01.159557 15915 webserver.cc:460] Webserver started at http://127.24.216.188:41511/ using document root <none> and password file <none>
I20260929 02:18:01.160274 15915 fs_manager.cc:359] Metadata directory not provided
I20260929 02:18:01.160362 15915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:18:01.160641 15915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:18:01.163727 15915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/data/instance:
uuid: "1f568764e12c46c7af48f9a6e014eab3"
format_stamp: "Formatted at 2026-09-29 02:18:01 on dist-test-slave-9k0h"
server_key: "8605c740c5f980e37e1f132661e8f3bb"
server_key_iv: "49638f04c3b8f67270c3a1044f33e2bd"
server_key_version: "encryptionkey@0"
I20260929 02:18:01.164482 15915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/wal/instance:
uuid: "1f568764e12c46c7af48f9a6e014eab3"
format_stamp: "Formatted at 2026-09-29 02:18:01 on dist-test-slave-9k0h"
server_key: "8605c740c5f980e37e1f132661e8f3bb"
server_key_iv: "49638f04c3b8f67270c3a1044f33e2bd"
server_key_version: "encryptionkey@0"
I20260929 02:18:01.169597 15915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 02:18:01.173089 15930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:18:01.174934 15915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:18:01.175112 15915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/data,/tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/wal
uuid: "1f568764e12c46c7af48f9a6e014eab3"
format_stamp: "Formatted at 2026-09-29 02:18:01 on dist-test-slave-9k0h"
server_key: "8605c740c5f980e37e1f132661e8f3bb"
server_key_iv: "49638f04c3b8f67270c3a1044f33e2bd"
server_key_version: "encryptionkey@0"
I20260929 02:18:01.175331 15915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:18:01.191267 15915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:18:01.196192 15915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:18:01.196455 15915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:18:01.223838 15915 rpc_server.cc:307] RPC server started. Bound to: 127.24.216.188:38227
I20260929 02:18:01.225631 15915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/data/info.pb
I20260929 02:18:01.226238 15981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.216.188:38227 every 8 connection(s)
I20260929 02:18:01.227725 25442 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu as pid 15915
I20260929 02:18:01.227895 25442 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGNPMkl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790647920678408-25442-0/minicluster-data/master-2/wal/instance
I20260929 02:18:01.228371 25442 external_mini_cluster.cc:1428] Setting key ac2fed6aefd3aac95435390c4bc2d991
I20260929 02:18:01.231011 15982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:18:01.237721 15982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:18:01 dist-test-slave-9k0h krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.188@KRBTEST.COM
Sep 29 02:18:01 dist-test-slave-9k0h krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648281, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.188@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
I20260929 02:18:01.249666 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.230333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.24.216.188:38227)
0929 02:18:01.231081 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.231093 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.231130 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:18:01.232667 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.232686 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.233046 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.233436 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.234860 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.236156 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.237396 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.238067 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.238321 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.242816 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:01.242823 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:01.242833 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:01.242866 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:01.245814 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.246544 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.246552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.246560 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.246725 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.247114 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.247119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.247124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.247622 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:01.247946 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:01.248361 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 02:18:01.248757 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":332,"threads_started":1}
I20260929 02:18:01.255245 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.240043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.190:41223 (local address 127.0.0.1:52104)
0929 02:18:01.240772 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:01.240817 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:18:01.240959 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:18:01.241946 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:18:01.241953 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:18:01.242730 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:18:01.243023 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:01.243032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.244286 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.244290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:01.245334 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:01.245345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.246082 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.246088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:01.246311 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.246889 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:18:01.246905 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:18:01.248974 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:18:01.252524 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:01.252538 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:01.252544 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:01.253112 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:01.253510 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:01.253513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:01.253515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:01.253634 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:01.254195 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:18:01.254203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:18:01.254376 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 02:18:01.254555 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 02:18:01.254852 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":324,"threads_started":1}
I20260929 02:18:01.255401 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.240157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52104 (local address 127.24.216.190:41223)
0929 02:18:01.241561 (+ 1404us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.241565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.241581 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:18:01.241626 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.241629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.241683 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.241820 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.243173 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.244102 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.245490 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.245898 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.246036 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.249157 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:01.249162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:01.249165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:01.249194 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:01.252309 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.253223 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.253228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.253232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.253394 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.253770 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.253774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.253779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.254031 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:01.254221 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:01.254660 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 02:18:01.254932 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1310,"thread_start_us":1164,"threads_started":1}
I20260929 02:18:01.256696 15982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:18:01 dist-test-slave-9k0h krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.190@KRBTEST.COM
Sep 29 02:18:01 dist-test-slave-9k0h krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648281, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.188@KRBTEST.COM for kudu/127.24.216.189@KRBTEST.COM
I20260929 02:18:01.267903 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.252800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.24.216.190:41223)
0929 02:18:01.253421 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.253428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.253451 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:18:01.253495 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.253499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.253561 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.253675 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.254911 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.256144 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.257740 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.258249 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.258386 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.262326 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:01.262339 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:01.262342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:01.262370 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:01.265263 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.265875 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.265882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.265885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.266008 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.266438 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.266443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.266446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.266711 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:01.266979 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:01.267273 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 02:18:01.267487 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":277,"threads_started":1}
I20260929 02:18:01.270586 15912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.257465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59108 (local address 127.24.216.189:43923)
0929 02:18:01.257632 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.257638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.257823 (+ 185us) server_negotiation.cc:407] Connection header received
0929 02:18:01.257981 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.257985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.258044 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.258146 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.258826 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.259616 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.260790 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.261429 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.261547 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.265004 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:01.265009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:01.265013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:01.265042 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:01.267675 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.268441 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.268449 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.268454 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.268614 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.269030 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.269034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.269036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.269324 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:01.269465 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:01.270007 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 02:18:01.270240 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 02:18:01.271514 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.257457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.189:43923 (local address 127.0.0.1:59108)
0929 02:18:01.257742 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:01.257755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 02:18:01.257870 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:18:01.258190 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:18:01.258194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:18:01.258546 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:18:01.258664 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:01.258671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.259775 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.259782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:01.260641 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:01.260651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.261698 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.261703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:01.261851 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.262410 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:18:01.262426 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:18:01.264818 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:18:01.267819 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:01.267824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:01.267830 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:01.268305 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:01.268726 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:01.268730 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:01.268732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:01.268916 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:01.269468 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:18:01.269473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:18:01.269654 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 02:18:01.270826 (+ 1172us) client_negotiation.cc:240] Negotiation successful
0929 02:18:01.271123 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260929 02:18:01.272739 15982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:18:01 dist-test-slave-9k0h krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.216.189@KRBTEST.COM
Sep 29 02:18:01 dist-test-slave-9k0h krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648281, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.188@KRBTEST.COM for kudu/127.24.216.188@KRBTEST.COM
I20260929 02:18:01.286979 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.273594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35368)
0929 02:18:01.273822 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:01.273834 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 02:18:01.273912 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:18:01.274241 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:18:01.274244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:18:01.274566 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:18:01.274730 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:01.274741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.275846 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.275852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:01.276768 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:01.276778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.277829 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.277833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:01.277979 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.278527 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:18:01.278550 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:18:01.280762 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:18:01.284090 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:01.284095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:01.284099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:01.284646 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:01.285117 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:01.285121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:01.285127 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:01.285276 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:01.285907 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:18:01.285912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:18:01.286062 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 02:18:01.286252 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 02:18:01.286559 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260929 02:18:01.287024 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.269604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.24.216.189:43923)
0929 02:18:01.270286 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.270290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.270309 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:18:01.270348 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.270351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.270404 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.270486 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.272098 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.273177 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.274730 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.275206 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.275304 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.280045 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:01.280051 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:01.280056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:01.280086 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:01.283960 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.284783 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.284788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.284792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.284952 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.285412 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.285418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.285422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.285786 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:01.286026 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:01.286355 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 02:18:01.286598 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":326,"threads_started":1}
I20260929 02:18:01.287189 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.273603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.24.216.188:38227)
0929 02:18:01.273756 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.273760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.273905 (+ 145us) server_negotiation.cc:407] Connection header received
0929 02:18:01.273986 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.273991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.274060 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.274180 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.274874 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.275712 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.276911 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.277668 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.277801 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.280911 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:01.280915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:01.280918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:01.280945 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:01.283936 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.284782 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.284788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.284792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.284964 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.285412 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.285418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.285422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.285773 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:01.285928 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:01.286366 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 02:18:01.286758 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":53}
I20260929 02:18:01.291808 25442 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:18:01.292452 15982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3: Bootstrap starting.
I20260929 02:18:01.296182 15982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3: Neither blocks nor log segments found. Creating new log.
I20260929 02:18:01.297473 15982 log.cc:824] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:18:01.294106 15809 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "eb6d3ff2e62043c8a9671be3aa27d819"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52108
I20260929 02:18:01.300575 15982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3: No bootstrap required, opened a new log
I20260929 02:18:01.307528 15982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } }
I20260929 02:18:01.307838 15982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f568764e12c46c7af48f9a6e014eab3, State: Initialized, Role: FOLLOWER
I20260929 02:18:01.308288 15982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } }
I20260929 02:18:01.309303 15991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } } }
I20260929 02:18:01.309695 15991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:18:01.310328 15982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:18:01.319620 16002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:18:01.319792 16002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:18:01.317675 15915 master_runner.cc:186] Error getting master registration for 127.24.216.190:41223: OK, instance_id { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" instance_seqno: 1790648280185406 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:18:01.325534 15915 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:18:01.326596 15834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } attempt: 4
Sep 29 02:18:01 dist-test-slave-9k0h krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648280, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.190@KRBTEST.COM for kudu/127.24.216.188@KRBTEST.COM
I20260929 02:18:01.340986 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.327138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35384)
0929 02:18:01.327440 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:01.327460 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:18:01.327583 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:18:01.327891 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:18:01.327897 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:18:01.328420 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:18:01.328632 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:01.328645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.329644 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.329650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:01.330700 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:01.330711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.331427 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.331433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:01.331652 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.332260 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:18:01.332276 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:18:01.334593 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:18:01.338571 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:01.338577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:01.338581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:01.338922 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:01.339290 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:01.339297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:01.339300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:01.339456 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:01.340016 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:18:01.340020 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:18:01.340188 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 02:18:01.340326 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 02:18:01.340592 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260929 02:18:01.340997 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.327236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.24.216.188:38227)
0929 02:18:01.327398 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.327402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.327514 (+ 112us) server_negotiation.cc:407] Connection header received
0929 02:18:01.327693 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.327700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.327758 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.327837 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.328773 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.329509 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.330846 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.331272 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.331403 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.334765 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:01.334772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:01.334776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:01.334816 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:01.338436 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.339040 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.339047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.339051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.339178 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.339562 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.339581 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.339584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.339897 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:01.340064 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:01.340424 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 02:18:01.340630 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 02:18:01.347239 15834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819: Bootstrap starting.
I20260929 02:18:01.351332 15834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819: Neither blocks nor log segments found. Creating new log.
I20260929 02:18:01.352617 15834 log.cc:824] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819: Log is configured to *not* fsync() on all Append() calls
I20260929 02:18:01.355942 15834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819: No bootstrap required, opened a new log
I20260929 02:18:01.363502 15834 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } }
I20260929 02:18:01.363900 15834 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6d3ff2e62043c8a9671be3aa27d819, State: Initialized, Role: FOLLOWER
I20260929 02:18:01.364408 15834 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } }
I20260929 02:18:01.365381 16005 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } } }
I20260929 02:18:01.365777 16005 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:18:01.366709 15834 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:18:01.366762 15809 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 02:18:01.367012 15809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:18:01.369443 15809 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } }
I20260929 02:18:01.372121 15957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1f568764e12c46c7af48f9a6e014eab3"
I20260929 02:18:01.372603 15957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:18:01.376264 15957 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6d3ff2e62043c8a9671be3aa27d819 in term 1.
I20260929 02:18:01.376633 15809 leader_election.cc:288] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [CANDIDATE]: Term 1 election: Requested vote from peers 6b2f1e5bdb9a43a7acaa454582727a17 (127.24.216.189:43923), 1f568764e12c46c7af48f9a6e014eab3 (127.24.216.188:38227)
I20260929 02:18:01.376899 15783 leader_election.cc:302] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f568764e12c46c7af48f9a6e014eab3, eb6d3ff2e62043c8a9671be3aa27d819; no voters:
I20260929 02:18:01.377264 16005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:18:01.397819 16005 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [term 1 LEADER]: Becoming Leader. State: Replica: eb6d3ff2e62043c8a9671be3aa27d819, State: Running, Role: LEADER
I20260929 02:18:01.398538 16005 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } }
I20260929 02:18:01.413914 15908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } attempt: 5
I20260929 02:18:01.422137 16012 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb6d3ff2e62043c8a9671be3aa27d819. Latest consensus state: current_term: 1 leader_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } } }
I20260929 02:18:01.422600 16012 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [sys.catalog]: This master's current role is: LEADER
I20260929 02:18:01.423885 16026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260929 02:18:01.425107 15788 tablet.cc:2369] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:18:01.448768 15936 tablet.cc:2369] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:18:01.450166 16026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:18:01.450304 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.409963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.24.216.190:41223)
0929 02:18:01.410383 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.410392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.410422 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:18:01.410564 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.410571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.410671 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.410805 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.412959 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.414079 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.433149 (+ 19070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.434227 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.434328 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.438969 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:01.438977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:01.438981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:01.439017 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:01.444182 (+ 5165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.447165 (+ 2983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.447171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.447175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.447355 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.448049 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.448067 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.448070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.448434 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:01.448683 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:01.449602 (+ 919us) server_negotiation.cc:299] Negotiation successful
0929 02:18:01.449848 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
Sep 29 02:18:01 dist-test-slave-9k0h krb5kdc[15754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790648280, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.216.189@KRBTEST.COM for kudu/127.24.216.188@KRBTEST.COM
I20260929 02:18:01.455519 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.412568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.24.216.188:38227)
0929 02:18:01.412737 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.412746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.418815 (+ 6069us) server_negotiation.cc:407] Connection header received
0929 02:18:01.419121 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.419130 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.419239 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.419329 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.427953 (+ 8624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.429329 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.431832 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.432573 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.432691 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.440051 (+ 7360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:01.440058 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:01.440062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:01.440099 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:01.445992 (+ 5893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.450840 (+ 4848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.450847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.450852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.451011 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.453625 (+ 2614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.453631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.453634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.453976 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:01.454301 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:01.454868 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 02:18:01.455098 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 02:18:01.458541 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.419376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59112 (local address 127.24.216.189:43923)
0929 02:18:01.419584 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.419591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.426816 (+ 7225us) server_negotiation.cc:407] Connection header received
0929 02:18:01.427323 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.427331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.427418 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.427497 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.430139 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.431500 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.433865 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.434747 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.434873 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.438776 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:01.438785 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:01.438793 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:01.438842 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:01.443921 (+ 5079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.444747 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.444753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.444757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.444897 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.445355 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.445360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.445364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.445694 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:01.445918 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:01.455989 (+ 10071us) server_negotiation.cc:299] Negotiation successful
0929 02:18:01.458171 (+ 2182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 02:18:01.462133 16024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.415304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.24.216.188:38227)
0929 02:18:01.433878 (+ 18574us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.433884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.433908 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:18:01.433955 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.433961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.434026 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.436768 (+ 2742us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.437070 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.438419 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.447556 (+ 9137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.448257 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.450670 (+ 2413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.454071 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:18:01.454078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:18:01.454083 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:18:01.454119 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:18:01.458362 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.459099 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.459106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.459130 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.459413 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:18:01.460384 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:18:01.460388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:18:01.460391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:18:01.460745 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:18:01.460887 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:01.461478 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 02:18:01.461693 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18434,"thread_start_us":19131,"threads_started":1}
I20260929 02:18:01.462417 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:01.414554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.216.188:38227 (local address 127.0.0.1:35404)
0929 02:18:01.414848 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 02:18:01.414868 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:18:01.414987 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:18:01.435851 (+ 20864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:18:01.435859 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:18:01.436380 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:18:01.436593 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:01.436606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.445880 (+ 9274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.445887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:01.447369 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:18:01.447384 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:18:01.450087 (+ 2703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:18:01.450094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:18:01.450367 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:01.451217 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:18:01.451239 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:18:01.453860 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:18:01.458476 (+ 4616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:01.458481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:01.458486 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:01.458967 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:01.459559 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:18:01.459565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:18:01.459580 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:18:01.460236 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:18:01.460905 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:18:01.460910 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:18:01.461126 (+ 216us) client_negotiation.cc:769] Sending connection context
0929 02:18:01.461382 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 02:18:01.461792 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260929 02:18:01.468647 15908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17: Bootstrap starting.
I20260929 02:18:01.472785 15908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17: Neither blocks nor log segments found. Creating new log.
I20260929 02:18:01.474112 15908 log.cc:824] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17: Log is configured to *not* fsync() on all Append() calls
I20260929 02:18:01.444123 15767 master_runner.cc:186] Error getting master registration for 127.24.216.189:43923: OK, instance_id { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" instance_seqno: 1790648280691124 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:18:01.476202 15767 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:18:01.479126 15908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17: No bootstrap required, opened a new log
I20260929 02:18:01.501525 15957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3 [term 1 FOLLOWER]: Refusing update from remote peer eb6d3ff2e62043c8a9671be3aa27d819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:18:01.502970 16005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:18:01.504102 15908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } }
I20260929 02:18:01.504509 15908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2f1e5bdb9a43a7acaa454582727a17, State: Initialized, Role: FOLLOWER
I20260929 02:18:01.505012 15908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } }
I20260929 02:18:01.506088 16031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } } }
I20260929 02:18:01.506675 16031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:18:01.524333 15908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:18:01.524750 15882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:18:01.525727 15991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb6d3ff2e62043c8a9671be3aa27d819. Latest consensus state: current_term: 1 leader_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } } }
I20260929 02:18:01.526175 15991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:18:01.528780 15882 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17 [term 1 FOLLOWER]: Refusing update from remote peer eb6d3ff2e62043c8a9671be3aa27d819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:18:01.529784 16005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:18:01.538079 16032 mvcc.cc:205] Tried to move back new op lower bound from 7334495361019674624 to 7334495360615202816. Current Snapshot: MvccSnapshot[applied={T|T < 7334495361019674624}]
I20260929 02:18:01.538727 15883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6b2f1e5bdb9a43a7acaa454582727a17"
I20260929 02:18:01.544102 16012 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } } }
I20260929 02:18:01.544373 16012 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [sys.catalog]: This master's current role is: LEADER
I20260929 02:18:01.547888 16031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb6d3ff2e62043c8a9671be3aa27d819. Latest consensus state: current_term: 1 leader_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } } }
I20260929 02:18:01.548187 16031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:18:01.549401 15991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } } }
I20260929 02:18:01.549645 15991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:18:01.550016 16026 catalog_manager.cc:1347] Generated new cluster ID: 75a18a5f064c4d89bb36310ce1f5184e
I20260929 02:18:01.550185 16026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:18:01.551681 15883 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eb6d3ff2e62043c8a9671be3aa27d819 for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260929 02:18:01.555624 16005 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } } }
I20260929 02:18:01.556586 16031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6d3ff2e62043c8a9671be3aa27d819" member_type: VOTER last_known_addr { host: "127.24.216.190" port: 41223 } } peers { permanent_uuid: "6b2f1e5bdb9a43a7acaa454582727a17" member_type: VOTER last_known_addr { host: "127.24.216.189" port: 43923 } } peers { permanent_uuid: "1f568764e12c46c7af48f9a6e014eab3" member_type: VOTER last_known_addr { host: "127.24.216.188" port: 38227 } } }
I20260929 02:18:01.556818 16031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:18:01.557292 16005 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819 [sys.catalog]: This master's current role is: LEADER
I20260929 02:18:01.559685 16036 mvcc.cc:205] Tried to move back new op lower bound from 7334495361019674624 to 7334495360615202816. Current Snapshot: MvccSnapshot[applied={T|T < 7334495361019674624}]
I20260929 02:18:01.585903 16026 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:18:01.586870 16050 catalog_manager.cc:1259] Loaded cluster ID: 75a18a5f064c4d89bb36310ce1f5184e
I20260929 02:18:01.587009 16050 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17: loading cluster ID for follower catalog manager: success
I20260929 02:18:01.587760 16026 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:18:01.594612 16050 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17: acquiring CA information for follower catalog manager: success
W20260929 02:18:01.595974 16050 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260929 02:18:01.603035 16026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb6d3ff2e62043c8a9671be3aa27d819: Generated new TSK 0
I20260929 02:18:01.604101 16026 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:18:01.764976 15988 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:18:01.739423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.24.216.190:41223)
0929 02:18:01.739642 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.739652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.761886 (+ 22234us) server_negotiation.cc:407] Connection header received
0929 02:18:01.763188 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.763195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.763312 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.763421 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.764638 (+ 1217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52122: BlockingRecv error: recv got EOF from 127.0.0.1:52122 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 02:18:01.769048 16024 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:18:01.746081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.24.216.188:38227)
0929 02:18:01.746301 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.746310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.765631 (+ 19321us) server_negotiation.cc:407] Connection header received
0929 02:18:01.765902 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.765908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.766020 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.766103 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.768805 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35408: BlockingRecv error: recv got EOF from 127.0.0.1:35408 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 02:18:01.771329 15989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:18:01.760212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.24.216.189:43923)
0929 02:18:01.760411 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.760418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.769833 (+ 9415us) server_negotiation.cc:407] Connection header received
0929 02:18:01.769989 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.769994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.770089 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.770243 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.771049 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59116: BlockingRecv error: recv got EOF from 127.0.0.1:59116 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 02:18:01.819351 15989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:18:01.816439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.24.216.189:43923)
0929 02:18:01.816655 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.816662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.817178 (+ 516us) server_negotiation.cc:407] Connection header received
0929 02:18:01.817538 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.817561 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.817702 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.817791 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.818993 (+ 1202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59126: BlockingRecv error: recv got EOF from 127.0.0.1:59126 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 02:18:01.867993 15989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:18:01.858522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.24.216.189:43923)
0929 02:18:01.858821 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.858828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.865866 (+ 7038us) server_negotiation.cc:407] Connection header received
0929 02:18:01.866433 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.866440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.866554 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.866652 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.867700 (+ 1048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59136: BlockingRecv error: recv got EOF from 127.0.0.1:59136 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260929 02:18:01.922102 15989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:18:01.912971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.24.216.189:43923)
0929 02:18:01.913216 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.913238 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.920228 (+ 6990us) server_negotiation.cc:407] Connection header received
0929 02:18:01.920695 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.920702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.920797 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.920871 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.921841 (+ 970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59142: BlockingRecv error: recv got EOF from 127.0.0.1:59142 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 02:18:01.979874 15989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:18:01.963895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.24.216.189:43923)
0929 02:18:01.964120 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:01.964127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:01.977992 (+ 13865us) server_negotiation.cc:407] Connection header received
0929 02:18:01.978261 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:01.978267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:01.978368 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:01.978464 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:01.979607 (+ 1143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59156: BlockingRecv error: recv got EOF from 127.0.0.1:59156 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260929 02:18:02.007414 15989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:18:02.005009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.24.216.189:43923)
0929 02:18:02.005258 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:02.005268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:02.005886 (+ 618us) server_negotiation.cc:407] Connection header received
0929 02:18:02.006122 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:02.006128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:02.006280 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:02.006394 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:02.007167 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59164: BlockingRecv error: recv got EOF from 127.0.0.1:59164 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 02:18:02.029408 15989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:18:02.026883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.24.216.189:43923)
0929 02:18:02.027171 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:02.027181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:02.027673 (+ 492us) server_negotiation.cc:407] Connection header received
0929 02:18:02.027901 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:02.027906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:02.027987 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:02.028123 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:02.029071 (+ 948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59168: BlockingRecv error: recv got EOF from 127.0.0.1:59168 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 02:18:02.067234 15989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:18:02.064841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.24.216.189:43923)
0929 02:18:02.065073 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:02.065083 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:02.065627 (+ 544us) server_negotiation.cc:407] Connection header received
0929 02:18:02.065865 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:02.065871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:02.065955 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:02.066089 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:02.066973 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59172: BlockingRecv error: recv got EOF from 127.0.0.1:59172 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 02:18:02.136754 15989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:18:02.134555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59176 (local address 127.24.216.189:43923)
0929 02:18:02.134809 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:02.134817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:02.135193 (+ 376us) server_negotiation.cc:407] Connection header received
0929 02:18:02.135407 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:02.135411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:02.135502 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:02.135595 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:02.136478 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59176: BlockingRecv error: recv got EOF from 127.0.0.1:59176 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 02:18:02.270718 15989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:18:02.268732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.24.216.189:43923)
0929 02:18:02.268977 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:02.268985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:02.269291 (+ 306us) server_negotiation.cc:407] Connection header received
0929 02:18:02.269547 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:02.269552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:02.269623 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:02.269736 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:02.270524 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59178: BlockingRecv error: recv got EOF from 127.0.0.1:59178 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 02:18:02.321777 16002 catalog_manager.cc:1259] Loaded cluster ID: 75a18a5f064c4d89bb36310ce1f5184e
I20260929 02:18:02.321877 16002 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3: loading cluster ID for follower catalog manager: success
I20260929 02:18:02.324586 16002 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3: acquiring CA information for follower catalog manager: success
I20260929 02:18:02.326450 16002 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1f568764e12c46c7af48f9a6e014eab3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 02:18:02.533748 15989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:18:02.531907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.24.216.189:43923)
0929 02:18:02.532171 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:02.532178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:02.532393 (+ 215us) server_negotiation.cc:407] Connection header received
0929 02:18:02.532636 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:02.532641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:02.532720 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:02.532833 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:18:02.533556 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59182: BlockingRecv error: recv got EOF from 127.0.0.1:59182 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 02:18:02.598618 16050 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b2f1e5bdb9a43a7acaa454582727a17: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:18:03.063735 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:03.056033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.24.216.189:43923)
0929 02:18:03.056480 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:03.056485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:03.056594 (+ 109us) server_negotiation.cc:407] Connection header received
0929 02:18:03.056787 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:03.056791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:03.056949 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:03.057106 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:18:03.057633 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:03.058439 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:03.060763 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:03.061446 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:03.061592 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:03.062279 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:18:03.063004 (+ 725us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:18:03.063149 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:03.063298 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 02:18:03.063398 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":245,"threads_started":1}
I20260929 02:18:03.075254 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:03.069338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.24.216.188:38227)
0929 02:18:03.070513 (+ 1175us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:03.070519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:03.070535 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:18:03.070622 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:03.070626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:03.070771 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:03.070900 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:18:03.071379 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:03.072085 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:03.073314 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:03.073693 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:03.073783 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:03.074286 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:18:03.074775 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:18:03.074878 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:03.074965 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 02:18:03.075028 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1073,"thread_start_us":977,"threads_started":1}
I20260929 02:18:03.096138 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:03.080720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.24.216.189:43923)
0929 02:18:03.080918 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:03.080928 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:03.089041 (+ 8113us) server_negotiation.cc:407] Connection header received
0929 02:18:03.089242 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:03.089251 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:03.089373 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:03.089498 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:18:03.090189 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:03.091589 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:03.093942 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:03.094578 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:03.094699 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:03.095392 (+ 693us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:18:03.095496 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:18:03.095608 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:03.095760 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 02:18:03.095838 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:18:03.099399 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:03.080695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.24.216.190:41223)
0929 02:18:03.092940 (+ 12245us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:03.092948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:03.092969 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:18:03.093077 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:03.093082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:03.093274 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:03.093449 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:18:03.094215 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:03.095095 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:03.097111 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:03.097729 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:03.097852 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:03.098416 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:18:03.098766 (+ 350us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:18:03.098886 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:03.099009 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 02:18:03.099094 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12102,"thread_start_us":12843,"threads_started":1}
I20260929 02:18:03.100067 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:18:03.080877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.24.216.188:38227)
0929 02:18:03.081070 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:18:03.081078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:18:03.094195 (+ 13117us) server_negotiation.cc:407] Connection header received
0929 02:18:03.094445 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:18:03.094450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:18:03.094533 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:18:03.094625 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:18:03.095238 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:03.096349 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:03.098258 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:18:03.098660 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:18:03.098750 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:18:03.099329 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:18:03.099426 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:18:03.099528 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:18:03.099684 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 02:18:03.099770 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:18:03.103819 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 15767
I20260929 02:18:03.104823 15829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:18:03.215267 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 15767
I20260929 02:18:03.241024 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 15841
I20260929 02:18:03.242111 15903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:18:03.349123 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 15841
I20260929 02:18:03.369402 25442 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu pid 15915
I20260929 02:18:03.370528 15977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:18:03.478139 25442 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGNPMkl/build/asan/bin/kudu with pid 15915
2026-09-29T02:18:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5117 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9598 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (362762 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST